Skip to content

feat(node): per-IP write-surface rate brake + purge-spam admin tool - #196

Merged
beardthelion merged 75 commits into
mainfrom
feat/write-sink-brake-coverage
Nov 13, 2026
Merged

beardthelion merged 75 commits into
mainfrom
feat/write-sink-brake-coverage

Conversation

@beardthelion

@beardthelion beardthelion commented Jul 13, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

Adds a per-IP rate brake on the authenticated non-creation write surface and a purge-spam admin subcommand for cleaning up an empty spam burst.

Motivation & context

Creation-surface abuse resistance. Before this, the write route groups had no shared per-IP limit, so a single-source flood on them was uncounted.

No linked issue.

Kind of change

  • Bug fix
  • Feature
  • Security fix
  • Docs
  • Tests / CI
  • Refactor (no behavior change)
  • Breaking or protocol change (issue required first)

What changed

Crate touched: gitlawb-node.

Two related pieces of the creation-surface abuse-resistance work:

  1. A per-IP write-surface rate brake. Every authenticated non-creation write sink now sits behind a per-client-IP flood brake on its own bucket: pull/issue comments, issue close, labels, stars, merges, protect/unprotect, replicas, webhook create/delete, visibility (including the GET on that group), agent deregister, tasks, bounties, profile, and every HTTP /graphql request. Issue comment and issue close debit the bucket inside the handler, after the read gate, rather than in router middleware; issue close also draws on its own GITLAWB_CLOSE_ISSUE_RATE_LIMIT bucket, and a close refused by either cap spends nothing from the other. Previously these route groups (the webhook routes among them) mounted with no shared write limit (issue close had only its own bucket), so a single-source flood on any of them was uncounted by the write budget (only repo/agent creation, git-push, and peer-sync were braked). The bucket is configured by GITLAWB_WRITE_RATE_LIMIT (default 600/hr, 0 disables), keyed on the resolved client IP via the existing GITLAWB_TRUSTED_PROXY boundary.

  2. A purge-spam admin subcommand for cleaning up a specific empty spam-burst that predates the mirror bot. Dry-run by default; --execute deletes.

Key decisions

  • Separate bucket from creation, not shared: a write flood cannot drain the repo-create budget and vice versa (same rationale as the existing sync_trigger vs peer_write split). A shared bucket would also self-throttle a legitimate session doing repo + issues + comments.
  • IP-only, no per-DID on the write groups: a did:key farm never trips a per-DID limit (the same reason git-receive-pack deliberately omits it), and a per-DID brake would false-positive busy legitimate agents.
  • GraphQL POST braked at the HTTP layer. An HTTP-layer brake cannot tell a query POST from a mutation POST, so it counts every /graphql request; /graphql/ws subscriptions stay unbraked. MutationRoot was the largest previously-uncounted write vector.
  • purge-spam safety: per-repo empty verification (zero git refs, with a bare-repo-marker guard so git's upward repo discovery can't read a parent checkout's refs and delete a live repo), a hard exclusion gate evaluated before the empty check, a re-verify immediately before delete (TOCTOU), and on-disk dir removal so DB and disk stay consistent. Dry-run is the default and prints the candidate list with ref-count evidence.

How a reviewer can verify

cargo test -p gitlawb-node --lib
cargo fmt --all --check
cargo clippy --workspace --all-targets -- -D warnings
  • cargo test --workspace green; cargo fmt --all --check and cargo clippy --workspace --all-targets -D warnings clean.
  • Real-node deny path driven through a client: a write sink returns 429 before signature verification (brake outermost), and under-limit writes reach the handler.
  • Adversarial TrustedProxy coverage: a spoofed leftmost X-Forwarded-For hop cannot rotate the bucket key, distinct trusted hops get distinct buckets, and an absent header falls back to the socket peer.
  • The load-bearing tests were RED-verified (removing the brake, the exclusion gate, the bare-repo guard, or the SQL normalization each turns the relevant test red).

Before you request review

  • Scope is one logical change; no unrelated churn
  • cargo test --workspace passes locally
  • New behavior is covered by tests (required for fixes)
  • cargo fmt --all and cargo clippy --workspace --all-targets -- -D warnings are clean
  • Commit titles use Conventional Commits (feat(...), fix(...), docs(...))
  • Docs / .env.example updated if behavior or config changed (or N/A)
  • Checked existing PRs so this isn't a duplicate

Protocol & signing impact

N/A: no change to DID/did:key, signatures, UCAN, ref certs, or P2P wire formats.

Notes for reviewers

This is the write-surface brake coverage plus the burst cleanup. iCaptcha enforcement, storage/size quotas, and moving the gate toward enforce-by-default are separate follow-ups, not in this PR.

Summary by CodeRabbit

  • New Features
    • Added a purge-spam admin command with a dry-run mode and optional execution.
    • Added configurable per-IP rate limiting for authenticated write requests and GraphQL HTTP requests. Set GITLAWB_WRITE_RATE_LIMIT=0 to disable it; GraphQL WebSocket traffic is unaffected.
  • Bug Fixes
    • Improved repository creation and fork handling under concurrent requests, including cleanup after failed forks.
    • Improved issue-close and pull-request merge handling when durable storage operations fail.
    • Rate-limit responses now provide an accurate Retry-After value.
  • Documentation
    • Documented the write-rate limit, IP handling, and related behavior in .env.example.
  • Tests
    • Expanded coverage for purge safety, rate limiting, concurrency, and storage failures.

…utes

Introduces a dedicated write_rate_limiter bucket (GITLAWB_WRITE_RATE_LIMIT,
default 600/hr, 0 disables), separate from the creation brake so a write flood
cannot drain the creation budget (KTD-1). Attaches rate_limit_by_ip to
write_routes, mirroring creation_routes. Per-DID deliberately not paired (a DID
farm never trips it).

Tests (route-level, execution-verified against Postgres): a write_routes sink
(PUT /star) is 429'd before auth; an exhausted write bucket does not throttle
repo creation (separate-bucket property).
Attaches rate_limit_by_ip (write bucket) to task_write_routes,
issue_write_routes, bounty_write_routes, profile_write_routes, and the /graphql
POST/GET surface (KTD-5 — an HTTP-layer brake counts every /graphql request;
/graphql/ws subscriptions stay unbraked). write_ip_limiter is now built once and
shared. Every non-git, non-creation authenticated write group is braked.

Tests (execution-verified against Postgres): /graphql POST is 429'd, and a
representative REST write sink (issue comment) is 429'd, by the write brake.
…rake

Middleware-level tests complementing the client_key unit coverage: a spoofer
varying the client-controlled leftmost X-Forwarded-For hop cannot rotate its
bucket key (keyed on the trusted rightmost hop); distinct trusted client hops
get distinct buckets (no cross-starvation); and a request with no trusted header
in a proxy mode falls back to the socket-peer key — the documented collapse,
asserted by name so any change is deliberate.

Operator-warning on missing-header deferred deliberately: a per-request warning
is a log-noise risk and the security-load-bearing fallback (peer key, never
skip) is what these tests lock down.
Adds a purge-spam subcommand (clap #[command(subcommand)] split; no-subcommand
path still runs the node unchanged). Candidate selection is signature-based, not
per-DID: a repo qualifies only if owned by the named burst DID AND verified empty
(zero git refs) per repo. A hard exclusion gate runs BEFORE the empty check, so
an empty repo owned by the content-bearing user or the mirror-bot DID is never a
candidate. Dry-run is the default; deletion needs an explicit --execute flag and
operates per-repo, never 'all repos of DID X'.

Tests (RED->GREEN, exclusion gate confirmed load-bearing by targeting an excluded
DID directly): empty target listed; target-with-refs spared; empty excluded/
intern repos spared (must-assert); dry-run deletes nothing; execute removes only
the empty target on disk.
P1 from code review: on_disk_ref_count shelled 'git for-each-ref' with only the
repo path as cwd and no --git-dir, so a path that exists but is not a bare repo
let git discover a parent .git (repos_dir may sit inside the operator's checkout)
and read a DIFFERENT repo's refs — possibly 0, deleting a real repo. Now require
the bare-repo markers (HEAD file + objects/ dir) before trusting any count;
anything else fails closed (skipped).

Tests: a non-git dir under a git ancestor (zero-ref) now fails closed — verified
RED without the guard (read the ancestor's 0 refs); target-DID-never-excluded
invariant pinned; write_flood test anchored to a genuinely-drained bucket so its
assert_ne can't pass vacuously.
…rage

Addresses the remaining code-review findings:
- purge-spam DID matching now uses did:key normalization on both sides
  (list_repos_by_owner_did via OWNER_KEY_CASE_SQL, is_excluded and the target
  scope via normalize_owner_key), so short/full form is consistent and an
  excluded identity is protected in either form. Adds a short-form exclusion test.
- Extract the per-IP write brake into a chainable WriteBraked helper so a future
  write group can't silently ship unbraked (kept separate from add_auth_layers
  by design; documented why).
- Mark --repos-dir / --database-url global so admin subcommands accept them after
  the subcommand (was an 'unexpected argument' parse error); verified.
- Document GITLAWB_WRITE_RATE_LIMIT in .env.example incl. the per-IP NAT-aggregate
  caveat so operators can size it.
- Coverage: adoption-floor (under-limit write passes), write-limit=0 disables,
  task/bounty/profile route-level 429s, /graphql/ws stays unbraked.

Full suite 927 pass / 0 fail; fmt + clippy -D warnings clean.
- purge-spam execute now re-verifies emptiness immediately before delete (TOCTOU:
  a push between selection and delete no longer risks deleting a now-non-empty
  repo) and removes the on-disk bare repo dir so DB and disk stay consistent; a
  per-repo delete failure warns and continues rather than aborting the batch.
- Tests (RED-verified where they cover new behavior): partition_for_delete skips
  a repo no longer empty; execute removes the on-disk dir (RED without the
  removal); list_repos_by_owner_did finds a short-form burst row (RED without the
  SQL normalization); every braked write group passes under-limit (per-group
  grant path, not just the star representative).
@coderabbitai

coderabbitai Bot commented Jul 13, 2026 •

Copy link
Copy Markdown
Contributor

Explain Change →

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review
Walkthrough

The pull request adds a purge-spam admin command, repository and database locking for purge and write operations, and a configurable per-IP limiter for authenticated REST writes and GraphQL HTTP requests. It also updates selected write handlers to account for guard-release and upload outcomes.

Changes: Spam purge administration

Layer / File(s) Summary
Purge command and storage contracts
crates/gitlawb-node/src/{config,main}.rs, crates/gitlawb-node/src/git/{repo_store,tigris}.rs, crates/gitlawb-node/Cargo.toml
Adds the purge-spam command and its dispatcher. Repository storage adds lock and archive operations used by the purge runner.
Candidate selection and database deletion
crates/gitlawb-node/src/{admin,db/mod}.rs
Selects candidates using normalized owner DIDs and exclusions. Database deletion handles repository-related rows and serializes deletion with writes to normalized slugs.
Locked purge execution and validation
crates/gitlawb-node/src/admin.rs
Adds dry-run and execute flows. Execution rechecks candidates under locks and tracks database, disk, and archive outcomes. Tests cover selection, path safety, locking, and object-store behavior.
Slug locking for deletion and ingestion
crates/gitlawb-node/src/db/mod.rs
Uses normalized slug locks for mirror ingestion and writes to slug-keyed tables. Integration tests cover deletion alongside concurrent updates.

Repository storage and write lifecycle

Layer / File(s) Summary
Repository creation and fork locking
crates/gitlawb-node/src/api/repos.rs
Creation and fork operations hold target repository locks through their writes. Fork cloning now has a configured timeout and removes the destination after clone failures.
Write guard release and error handling
crates/gitlawb-node/src/api/{issues,pulls}.rs, crates/gitlawb-node/src/error.rs, crates/gitlawb-node/src/git/store.rs
Issue close documents its rollback behavior on failed upload. Pull-request merge checks the guard-release result before updating PR state. The remaining changes adjust test imports and error comments.

Authenticated write rate limiting

Layer / File(s) Summary
Limiter configuration and state
.env.example, crates/gitlawb-node/src/{main,state,auth/mod,test_support}.rs
Adds the hourly write-limit setting, initializes and cleans up the write limiter, and updates fixtures.
Limiter decisions and retry delays
crates/gitlawb-node/src/rate_limit.rs
Returns computed rejection delays and uses them in 429 responses. Tests cover trusted-proxy IP selection and Retry-After behavior.
Write limiter route wiring
crates/gitlawb-node/src/server.rs, crates/gitlawb-node/src/api/repos.rs
Applies per-IP limiting to authenticated REST write groups and all GraphQL HTTP requests. GraphQL WebSocket requests remain outside the limiter. Route tests cover buckets, disabled limiting, and WebSocket behavior.

Estimated code review effort: 5 (Critical) | ~100 minutes

Change: Feature

Sequence Diagram(s)

sequenceDiagram
  participant AdminCLI
  participant Main
  participant PurgeRunner
  participant Db
  participant RepoStore
  AdminCLI->>Main: PurgeSpam command
  Main->>Db: connect and list repositories
  Main->>RepoStore: initialize archive and lock access
  Main->>PurgeRunner: run purge
  PurgeRunner->>RepoStore: lock repository and refresh archive
  PurgeRunner->>Db: delete repository data
  PurgeRunner->>RepoStore: remove local directory and archive
Loading
sequenceDiagram
  participant Client
  participant Router
  participant WriteLimiter
  participant WriteRoute
  Client->>Router: authenticated write request
  Router->>WriteLimiter: resolve client IP and check bucket
  WriteLimiter->>WriteRoute: forward permitted request
Loading

Possibly related issues

  • Twigpine/node#563: The pull request leaves GraphQL WebSocket requests outside the write limiter; the issue concerns authenticated mutations on that surface.

Possibly related PRs

  • Twigpine/node#54: Changes to guard release and durable-upload outcomes overlap with the issue and pull-request write handling in this pull request.

Suggested reviewers: kevincodex1

Merge Risk: Moderate · previous review da876

The purge command does not work with its default repository path. When run with --execute, it can leave orphaned rows or archives, or delete a newly published archive. The new write rate limit can be bypassed or exhausted by unrelated traffic. Fork and lock handling can leave partial clones or stuck locks. Fix the confirmed purge, limiter and fork issues before merging.

Pre-merge checks: Passed 4

Check name Status Explanation
Title check Passed The title clearly and concisely identifies both primary changes: the per-IP write-surface rate brake and the purge-spam admin tool.
Linked Issues check Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check Passed Check skipped because no linked issues were found for this pull request.
Description check Passed The description follows the template and covers the Summary, Motivation, Kind of change (Feature), What changed, verification commands, Before-you-request-review checklist, Protocol and signing impact…
Full details: Description check

Explanation

The description follows the template and covers the Summary, Motivation, Kind of change (Feature), What changed, verification commands, Before-you-request-review checklist, Protocol and signing impact, and Notes for reviewers. It states the scope of the write-surface rate brake and the purge-spam subcommand, and it names the GITLAWB_WRITE_RATE_LIMIT default and the safety gates for purge. Gaps: the Motivation section has no linked issue (the author states 'No linked issue'), which is acceptable for an optional link but leaves the 'Closes #' line unfilled. The How-a-reviewer-can-verify section gives commands but no repro steps for the purge-spam dry-run or the rate-limit deny path, and the checklist items are left unchecked. The description is mostly complete, so it passes.

Finishing Touches

Generate unit tests (beta)

  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

@beardthelion beardthelion added crate:node gitlawb-node — the serving node and REST API kind:feature New capability or surface subsystem:identity DID/UCAN, http-sig auth, push authorization labels Jul 13, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/gitlawb-node/src/main.rs (1)

440-455: 🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win

write_rate_limiter is missing from the periodic cleanup loop.

Every other limiter (rate_limiter, create_ip_rate_limiter, push_rate_limiter, sync_trigger_rate_limiter, peer_write_rate_limiter) is swept here, but the newly added write_rate_limiter is not. It is bounded, so no unbounded leak, yet stale keys are only evicted lazily on a capacity miss instead of proactively like its peers — an inconsistency worth closing.

🧹 Proposed fix
         let peer_write_rl = state.peer_write_rate_limiter.clone();
+        let write_rl = state.write_rate_limiter.clone();
         let db = state.db.clone();
                         peer_write_rl.cleanup().await;
+                        write_rl.cleanup().await;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/main.rs` around lines 440 - 455, Include the
state.write_rate_limiter in the periodic cleanup task alongside the other cloned
rate limiters. Clone it before spawning the task and invoke its cleanup method
in the five-minute tokio::select! interval, preserving the existing cleanup
behavior for all other limiters.
🧹 Nitpick comments (1)
crates/gitlawb-node/src/api/repos.rs (1)

2629-2966: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider extracting a shared test-request helper to reduce duplication.

All 9 new tests repeat the same Request::builder() + ConnectInfo insertion + oneshot() + status-assert boilerplate. rate_limit.rs in this same PR already factored this pattern into post_from/post_with helpers — a similar helper here (e.g., send_from(&router, method, uri, body, peer)) would cut most of the repetition across write_route_is_rate_limited_by_ip, write_flood_does_not_drain_creation_budget, graphql_post_is_rate_limited_by_ip, issue_comment_is_rate_limited_by_ip, under_limit_write_is_not_throttled, write_rate_limit_zero_disables_the_brake, task_bounty_profile_writes_are_rate_limited, graphql_ws_is_not_braked, and every_write_group_passes_under_limit.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/api/repos.rs` around lines 2629 - 2966, Reduce
duplicated request setup across the listed rate-limit tests by adding a shared
test helper such as send_from that accepts the router, HTTP method, URI, body,
and peer, inserts ConnectInfo, sends via oneshot, and returns the response or
status. Update each named test to use this helper while preserving its existing
request bodies, methods, URIs, and status assertions.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/gitlawb-node/src/main.rs`:
- Around line 440-455: Include the state.write_rate_limiter in the periodic
cleanup task alongside the other cloned rate limiters. Clone it before spawning
the task and invoke its cleanup method in the five-minute tokio::select!
interval, preserving the existing cleanup behavior for all other limiters.

---

Nitpick comments:
In `@crates/gitlawb-node/src/api/repos.rs`:
- Around line 2629-2966: Reduce duplicated request setup across the listed
rate-limit tests by adding a shared test helper such as send_from that accepts
the router, HTTP method, URI, body, and peer, inserts ConnectInfo, sends via
oneshot, and returns the response or status. Update each named test to use this
helper while preserving its existing request bodies, methods, URIs, and status
assertions.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: e186f361-729a-419c-b633-a90e12b77a44

📥 Commits

Reviewing files that changed from the base of the PR and between ad7c2b2 and 27b4a2a.

📒 Files selected for processing (11)
  • .env.example
  • crates/gitlawb-node/src/admin.rs
  • crates/gitlawb-node/src/api/repos.rs
  • crates/gitlawb-node/src/auth/mod.rs
  • crates/gitlawb-node/src/config.rs
  • crates/gitlawb-node/src/db/mod.rs
  • crates/gitlawb-node/src/main.rs
  • crates/gitlawb-node/src/rate_limit.rs
  • crates/gitlawb-node/src/server.rs
  • crates/gitlawb-node/src/state.rs
  • crates/gitlawb-node/src/test_support.rs

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

High — fix before merge

H1. Read/fork handlers run blocking git subprocesses directly on the tokio runtime (no spawn_blocking, no timeout)

  • fork_repo — crates/gitlawb-node/src/api/repos.rs:1567-1575 invokes std::process::Command::new("git").args(["clone","--mirror",…]).output() inside the async handler. output() blocks the calling worker thread; git clone --mirror of a non-trivial repo can take minutes and there is no timeout, unlike every other git walk in this file which correctly uses tokio::task::spawn_blocking (repos.rs:67, 110, 661, 1143) or tokio::process::Command (info/refs, upload-pack, receive-pack).
  • Read metadata handlers — list_commits/get_tree/get_blob call store::{resolve_head, log, ls_tree, read_file} (api/repos.rs:360-361, 397, 443, 481) which are synchronous std::process::Command::output() calls (crates/gitlawb-node/src/git/store.rs:92-231). A slow disk or large repo blocks a worker thread for seconds; with a finite worker pool a handful of concurrent /commits, /tree, /blob, or a single /fork can starve the whole runtime, including /health and /ready (same runtime in build_router).
  • Impact: availability/reliability regression, not just latency. git_service_timeout_secs is bypassed on these paths.
  • Fix: wrap the blocking store calls (and the fork clone) in tokio::task::spawn_blocking and bound with tokio::time::timeout(git_service_timeout_secs, …), returning AppError::Timeout.

H2. write_rate_limiter is never reaped by the background cleanup loop

  • crates/gitlawb-node/src/main.rs:440-455 — the periodic task clones exactly five limiters (rate_limiter, create_ip_rate_limiter, push_rate_limiter, sync_trigger_rate_limiter, peer_write_rate_limiter) and calls .cleanup() on each. state.write_rate_limiter (declared at main.rs:329-333, wired into AppState at :404) is omitted, even though it protects the entire authenticated non-creation write surface + /graphql (server.rs:84-98, 110, 154+).
  • The RateLimiter design makes the inline capacity sweep amortized and explicitly relies on the background loop for full reclamation (rate_limit.rs:36-39, 94-114). With no background sweep, write_rate_limiter only reclaims expired keys once its map hits the 200,000-key cap. Until then every distinct client IP persists as a stale Window for the full 1-hour window and beyond → steady memory growth (bounded at the cap) and a latent correctness cliff: once at cap, the amortized inline sweep becomes the only reclaimer and a distinct-IP write flood serializes behind O(max_keys) scans while legitimate new IPs get spuriously 429'd between sweeps.
  • Fix: add let write_rl = state.write_rate_limiter.clone(); and write_rl.cleanup().await; inside the cleanup tick beside the other five.

Medium — should fix

M1. fork_repo / create_repo orphan on-disk repos (and a spent proof / uploaded object) on late DB failure

  • fork_repo ordering (api/repos.rs:1555 → 1567 → 1585 → 1605): spend proof → git clone --mirror to disk_path → release_after_write (Tigris upload) → db.create_repo. If create_repo fails, the on-disk mirror and the already-uploaded Tigris object are left with no DB row, and the single-use iCaptcha proof is already consumed.
  • create_repo (api/repos.rs:208 → 210-219 → 236): proof.consume then repo_store.init (blocking git init) happen before db.create_repo. A DB failure orphans the freshly-initialized directory. init itself is also a blocking git call on the async runtime (same class as H1).
  • Fix: make disk materialization and proof spend adjacent to a guaranteed-successful DB write, and on db.create_repo failure best-effort remove_dir_all(disk_path) (and prefer deferring the Tigris upload until after the row commits).

M2. TrustedProxy IP trust is config-asserted, not network-enforced — header rotation defeats every per-IP brake

  • crates/gitlawb-node/src/rate_limit.rs:198-218 (client_key): in Fly/XForwardedFor modes the limiter key is taken entirely from Fly-Client-IP or the rightmost X-Forwarded-For hop, and only falls back to the socket peer when the header is absent. There is no check that the socket peer (ConnectInfo) is actually the operator's proxy. If GITLAWB_TRUSTED_PROXY is set but the node's HTTP port is reachable directly (misconfigured firewall, exposed Fly port, second ingress, internal attacker), a client can set Fly-Client-IP: <random> / append any rightmost X-Forwarded-For hop and get a fresh bucket per request, evading the creation/write/push/sync brakes (the only thing stopping a single-source spam/push flood). The unit tests cover header parsing but never assert "peer not the proxy → key on peer."
  • Fix: honor the forwarded header only when ConnectInfo peer is in a configured proxy CIDR/IP allowlist; otherwise key on the socket peer (fail closed). Add a test for "trust mode set, request from a non-proxy IP."
  • Note (validation): exploitability is deployment-dependent (Fly's edge does overwrite Fly-Client-IP), so rated Medium not Blocker.

M3. purge-spam --execute deletes via remove_dir_all on an unsanitized repo_name (path traversal → arbitrary repo deletion)

  • crates/gitlawb-node/src/git/store.rs:450-453 — repo_disk_path sanitizes owner_did (replace([':', '/'], "_")) but joins repo_name verbatim via format!("{repo_name}.git") with no ../slash/dot validation.
  • crates/gitlawb-node/src/admin.rs:243-244 calls std::fs::remove_dir_all(store::repo_disk_path(&repos_dir, &c.owner_did, &c.name)) on that path. The fail-closed marker guard (admin.rs:128) only requires a HEAD file + objects/ dir — i.e. any existing legitimate bare repo satisfies it. API names are validated (api/repos.rs:189-197), but smart_http_repo_name explicitly notes repos are "creatable via the peer mirror path, which skips API name validation" (api/repos.rs:490-494), so a burst-DID row with a ../ name can be planted, then deleted outside repos_dir.
  • Fix: validate repo_name (and re-validate owner_did) the same way the serving paths do (repo_store::validate_repo_name/validate_path_components), and assert the resolved path is canonically inside repos_dir before any remove_dir_all. Ideally make repo_disk_path itself reject unsafe names.
  • Note: a destructive remove_dir_all should never run on a DB-sourced, unsanitized name. End-to-end exploitability is narrow (requires the burst DID + marker check at the traversed path), so rated Low/Medium — but it is a clear defense-in-depth defect.

M4. purge-spam TOCTOU: emptiness re-checked once as a batch snapshot, not immediately before each delete — races a live push

  • crates/gitlawb-node/src/admin.rs:221-235 — partition_for_delete (:224-226) re-verifies emptiness for all candidates up front, producing to_delete; the actual delete_repo_by_id runs later in a loop (:235). Between the snapshot and a given delete, a concurrent git-receive-pack on the running node can add a ref, making the repo non-empty — yet it is still deleted (silent data loss). The comment claims it "re-verify[s] emptiness immediately before deleting," but the recheck is a single batch pass, not a per-candidate check immediately before delete_repo_by_id.
  • The execute path also does not acquire the node's acquire_write advisory lock (api/repos.rs:927), so it races a live push: a push can leave orphaned Tigris objects / ref-certificate rows referencing a now-deleted repos.id, and remove_dir_all can race an in-flight receive-pack writing objects/.
  • Fix: re-verify ref_count_on_disk immediately before each delete_repo_by_id (or wrap recheck+delete of each repo in a repo_store.acquire_write guard / DB transaction re-checking a ref signal). Document that the tool must be run against a quiesced node for the target DID.

M5. iCaptcha gate fails open in enforce mode when the public key cannot load

  • crates/gitlawb-node/src/icaptcha.rs:302-304 — inside decide, if v.key.is_none() { return Decision::Allow; } runs before the mode match, so in Mode::Enforce a missing key (iCaptcha unreachable at startup, DNS/BGP block, bad ICAPTCHA_PUBKEY) allows every create/registration request with no proof. init() (icaptcha.rs:160-166) leaves key = None and only warns. Since the gate defaults to off, the default deployment already has no captcha; enforce-mode is now trivially bypassable by denying the key fetch, directly weakening the abuse protection this PR leans on after PoW removal.
  • Fix: in enforce mode fail closed (reject) when key.is_none(); or refuse to start in enforce mode without a key. At minimum document that enforce+no-key == fail-open.

M6. purge-spam deletes the DB row, then removes the dir; a disk failure is swallowed while the deleted counter is already incremented

  • crates/gitlawb-node/src/admin.rs:235-248 — on Ok(n) the row is counted as deleted before remove_dir_all, which on error only warn!s. The final "deleted N repo row(s)" summary therefore reports success even when the on-disk repo survives → DB/disk drift and a misleading operator-facing count. No retry / non-zero exit signal.
  • Fix: remove the on-disk dir first (or track disk-failure separately from the DB-deleted count) and surface "N deleted, M failed."

M7. delete_repo_by_id (and purge-spam) leave dependent rows orphaned

  • crates/gitlawb-node/src/db/mod.rs:1288-1294 — DELETE FROM repos WHERE id = $1 with no cascade. Child tables keyed on repo_id (branch_cids, ref_certificates, visibility_rules, repo_stars, encrypted_blobs, received_ref_updates, repo_replicas, arweave_anchors, webhooks, pull_requests) keep orphan rows that accumulate and can be re-joined if a repo with the same id is later created.
  • Fix: add ON DELETE CASCADE (or explicit child deletes) for the repo_id FKs, ideally within the same transaction as the row delete.

Low / Info — nice to have

  • L1. Rate-limiter bounded-map DoS. rate_limit.rs:96-114 rejects new keys when at the 200k cap; a distinct-key flood (IPv6 variation or the M2 header rotation) then 429s legitimate new clients for up to a full window. Consider evicting expired entries to admit new keys, or a metric on cap-full rejections.
  • L2. Limiter state is per-process / in-memory (rate_limit.rs:24-41). In multi-machine Fly deployments the effective per-IP/per-DID limit is N × configured and a restart wipes buckets. Known tradeoff — document explicitly; consider shared counters for the spam-relevant paths.
  • L3. Non-atomic create_repo. api/repos.rs:203-236 check-then-insert with no UNIQUE(owner_did, name); the idx_repos_owner_key_name is a plain index (db/mod.rs:474-490). A concurrent create race surfaces a raw UNIQUE(disk_path) violation mapped to a generic 500 instead of RepoExists (409).
  • L4. Read handlers mask git errors as empty 200 OK. api/repos.rs:361 (store::log(...).unwrap_or_default()) and the get_tree siblings coerce real backend failures into empty lists, unlike get_blob (api/repos.rs:398-410) which classifies errors. Map Err to AppError::Git/NotFound.
  • L5. parse_ref_updates silently tolerates malformed/truncated bodies (api/repos.rs:1651-1699) → empty Vec<RefUpdate>; receive-pack then proceeds with zero ref updates (skipping branch-protection/owner-push checks). Return 400 on a non-empty body that parses short.
  • L6. Push bookkeeping errors swallowed (api/repos.rs:959, 977, 982, 999-1005 let _ = / only-warn!) — transient DB blips silently lose record_push/update_trust_score. At least raise log severity / add a durable outbox.
  • L7. git_info_refs receive-pack brake skipped when client_key returns None (api/repos.rs:556 / rate_limit.rs:198): if no trusted header and no ConnectInfo peer, the if let Some(key) guard is skipped and the request proceeds unthrottled. Unreachable in prod (socket peer always present) but a divergent contract from rate_limit_by_ip; document or reject instead of skip.
  • L8. Malformed rate-limit env vars silently fall back to defaults. main.rs:304-307, 325-328, 343-346 parse GITLAWB_CREATE/WRITE/PUSH_RATE_LIMIT with .ok().and_then(parse).unwrap_or(default); a fat-fingered value silently disables the intended (stricter) brake with no warning, unlike the clap-validated siblings. Warn (or fail fast) on present-but-unparseable values.
  • L9. .env.example docs drift. Missing GITLAWB_IPFS_API, GITLAWB_TIGRIS_BUCKET, GITLAWB_METRICS_ADDR, GITLAWB_SHUTDOWN_GRACE_SECS (parsed in config.rs:88,161,188,194). Reverse direction is clean. Add with defaults.
  • L10. Weak dry-run safety test. admin.rs:471-492 (dry_run_deletes_nothing) seeds repos with an empty repos_dir, so run_purge_spam hits the no-candidate early return before any delete logic — it does not exercise the if !execute { return } guard when candidates exist. Add a dry-run test with a real empty on-disk target repo so a regression that deletes in dry-run is caught.
  • L11. Unauthenticated /metrics (main.rs:828-875, config.rs:187 warns it is unauthenticated). If bound to a non-loopback addr it leaks peer/repo/push counts. Operator-controlled; note in hardening guidance.
  • L12. Git error strings returned to clients (AppError::Git(e.to_string()) sites, e.g. api/repos.rs:398-410). Can surface internal paths; return opaque messages, keep detail in server-side logs.

…ve_dir_all

A peer-mirror row skips API name validation, so a burst-owned repo can carry
a '../' name; repo_disk_path joined it verbatim and the emptiness marker check
passed on the traversed target, so purge-spam selected and remove_dir_all'd a
repo OUTSIDE repos_dir. Validate the name via repo_store::validate_repo_name at
selection (fail closed -> non-candidate) and assert canonical containment inside
repos_dir immediately before the destructive remove. Adversarial RED->GREEN: a
victim bare repo reachable via '../../victim' is deleted before, survives after;
symlink-escape covered by path_within.
delete_repo_by_id was a bare DELETE FROM repos with no cascade and the schema
has no FKs, so every child row (ref_certificates, pull_requests + pr_reviews/
pr_comments, webhooks, push_events, protected_branches, repo_stars, repo_replicas,
repo_labels, visibility_rules, encrypted_blobs, repo_icaptcha_proofs, agent_tasks,
and the slug-keyed branch_cids/sync_queue/received_ref_updates/arweave_anchors)
was orphaned and could re-join a later repo reusing the id. Wrap parent+child
deletes in one transaction, resolving the owner_short/name slug for the repo-TEXT
children. bounties (financial) and issue_comments (unmappable) are left intact by
design. RED->GREEN test seeds a repo_id child, a slug child, and a PR grandchild.
The PR's new write_rate_limiter was the one limiter omitted from main()'s
hand-maintained cleanup list, so its key map never shed expired entries until
the 200k cap. Move the sweep into AppState::cleanup_rate_limiters (co-located
with the limiter fields, single source of truth) and call it from the loop, so
a newly-added limiter is reaped rather than silently dropped. Guard test seeds a
reclaimable entry into write_rate_limiter and asserts the sweep reaps it — RED
if write_rate_limiter is dropped from cleanup_rate_limiters (verified).
purge-spam took no write lock, so a concurrent receive-pack could land a ref
between the emptiness recheck and the delete, silently losing a live push.
Add RepoStore::try_lock_repo — a lock-only, non-blocking counterpart to
acquire_write (no Tigris I/O) whose guard holds a dedicated pooled connection so
the advisory lock and its unlock run on the same session. purge now locks each
candidate, rechecks emptiness UNDER the lock, deletes, and releases; a repo held
by a live writer is skipped, not force-deleted. RED->GREEN: a locked empty repo
survives purge, then deletes once the writer releases (neuter-check confirms the
lock is load-bearing).
…etes

The Ok(n) arm incremented the deleted total before remove_dir_all, whose failure
only warn!'d, so the summary reported N deleted even when the on-disk dir
survived (DB/disk drift shown as clean success). run_purge_spam now returns a
PurgeSummary and counts a failed (or containment-refused) on-disk removal in
disk_failed, never folding it into deleted; the summary line surfaces
'M on-disk removal(s) failed'. RED->GREEN: a read-only parent dir forces the
removal to fail; deleted=1 AND disk_failed=1 with the dir surviving (neuter-check
confirms the counter is load-bearing).
…y defaulting

GITLAWB_CREATE/WRITE/PUSH_RATE_LIMIT parsed with .ok().and_then(parse).unwrap_or,
so a typo (e.g. 5O with a letter O) silently reverted to the default and left the
operator believing a stricter cap was in force. Route all three through
rate_limit_from_env, which distinguishes absent/empty (default, silent) from
present-but-unparseable (default, WARN). Pure resolve_rate_limit is unit-tested
across absent/empty/valid/zero/unparseable — 0 stays valid (its own ==0 warn
handles the disable case).
…DOWN_GRACE_SECS in .env.example

Four config knobs read in config.rs were absent from .env.example, so operators
had no template for them. Add each in its matching section with the config
default (INV-24: operator docs match the config the code reads, in the same PR
that touches the file).
…present

dry_run_deletes_nothing ran against an empty repos_dir, so select_spam_candidates
found nothing and run_purge_spam hit the no-candidate early return — the
if !execute { return } guard was never exercised with candidates present (a
vacuous guard, INV-21). Materialize a real empty bare repo so a genuine candidate
exists, then assert dry-run leaves both the row and the on-disk dir. Verified
load-bearing: removing the !execute guard turns the test RED (dry-run deletes).
@beardthelion

Copy link
Copy Markdown
Collaborator Author

Pushed the fixes at ffdc167. Every behavior change is test-first with the deny/failure case as the observed evidence; full suite 525 pass, clippy -D warnings and fmt clean.

Fixed in this PR

  • [H2] Reap write_rate_limiter in the cleanup loop — the PR's new limiter was the one omitted from the hand-maintained list, so its key map never shed until the 200k cap. Moved the sweep into AppState::cleanup_rate_limiters, co-located with the limiter fields (single source of truth). Guard test seeds a reclaimable entry and asserts the sweep reaps it; RED if write_rate_limiter is dropped from the sweep (verified). 4943430
  • [M3] Reject path-traversal repo names before remove_dir_all — a peer-mirror row skips API name validation, so a ../ name resolved onto a real bare repo outside repos_dir and got selected + deleted. Validate the name at selection (fail closed) via repo_store::validate_repo_name, and assert canonical containment right before the destructive remove. RED->GREEN: a victim repo outside repos_dir is deleted before, survives after; symlink-escape covered by a separate path_within test. 98150db
  • [M7] Delete child rows transactionally in delete_repo_by_id (db/mod.rs) — was a bare DELETE FROM repos with no FKs, orphaning ~14 child tables incl. PR grandchildren. Now one transaction deletes children (by repo_id and the owner_short/name slug) then the parent. bounties (financial) and issue_comments (no issues table to map) are intentionally left. RED->GREEN seeds a repo_id child, a slug child, and a PR review. 1688bdc
  • [M4] Hold the per-repo advisory lock across purge recheck+delete — added RepoStore::try_lock_repo, a lock-only counterpart to acquire_write (no Tigris I/O) whose guard holds a dedicated pooled connection so lock+unlock stay on one session. Purge locks each candidate, rechecks emptiness under the lock, deletes, releases; a repo held by a live writer is skipped. RED->GREEN: locked repo survives purge, deletes after release. 1305167
  • [M6] Count disk-removal failures separately from deletes — run_purge_spam returns a PurgeSummary; a failed (or containment-refused) on-disk removal is counted in disk_failed, never folded into deleted, so the summary can't report clean success while a dir survives. aab70ec
  • [L8] Warn on unparseable rate-limit env vars — GITLAWB_CREATE/WRITE/PUSH_RATE_LIMIT now distinguish absent/empty (silent default) from present-but-unparseable (default + warn), so a typo can't silently disable the intended cap. cecca67
  • [L9] Document GITLAWB_IPFS_API/TIGRIS_BUCKET/METRICS_ADDR/SHUTDOWN_GRACE_SECS in .env.example. f4d5bec
  • [L10] Strengthen the dry-run test to run with a real on-disk candidate present, so the !execute guard is actually exercised; RED if the guard is removed. 6da0cf5

Deferred to follow-up issues (pre-existing, outside this PR's diff)

store.rs, icaptcha.rs, and the fork/create ordering are untouched by this PR, so I split these rather than widen scope. Glad to fold any in here instead if you'd rather.

Declined

  • [L1] The bounded map already evicts expired keys then admits at the cap (rate_limit.rs:96-114, test expired_keys_evicted_to_admit_new_when_full); the reclaim gap is bounded by sweep_interval (<=1s under the 3600s window), not a full window. No change.

@jatmn ready for another look.

@beardthelion
beardthelion requested a review from jatmn July 15, 2026 03:38

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
.env.example (1)

143-152: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Document all GraphQL HTTP requests covered by this limiter.

The route wiring in crates/gitlawb-node/src/server.rs applies the brake to the whole /graphql route, so queries and the playground GET also consume this bucket—not only mutations. Update the text to say “GraphQL HTTP requests” or narrow the middleware.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.env.example around lines 143 - 152, Update the write rate-limiting
documentation near GITLAWB_WRITE_RATE_LIMIT to state that the limiter covers all
GraphQL HTTP requests on the /graphql route, including queries and the
playground GET, rather than describing only GraphQL mutations. Keep the existing
rate-limit scope and configuration details unchanged.
🧹 Nitpick comments (3)
.env.example (1)

19-19: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Keep the dotenv keys in linter order.

dotenv-linter reports that GITLAWB_METRICS_ADDR should appear before GITLAWB_PORT; moving it will keep configuration checks clean.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In @.env.example at line 19, Move the GITLAWB_METRICS_ADDR entry in .env.example
so it appears before GITLAWB_PORT, preserving the existing key definitions and
values.

Source: Linters/SAST tools

crates/gitlawb-node/src/main.rs (1)

292-345: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Extract the repeated per-IP limiter setup (create/write/push) into a helper.

The create/write/push blocks each repeat "resolve env limit → build RateLimiter::new_bounded → warn if zero" verbatim, just with different var names/messages. A small helper collapses the duplication and makes adding a fourth limiter trivial.

♻️ Proposed refactor
+fn build_ip_limiter(limit: usize, var: &str, label: &str) -> rate_limit::RateLimiter {
+    let limiter = rate_limit::RateLimiter::new_bounded(
+        limit,
+        std::time::Duration::from_secs(3600),
+        200_000,
+    );
+    if limit == 0 {
+        tracing::warn!("{var}=0 — per-IP {label} rate limiting disabled");
+    }
+    limiter
+}
+
     let create_limit = rate_limit_from_env("GITLAWB_CREATE_RATE_LIMIT", 120);
-    let create_ip_rate_limiter = rate_limit::RateLimiter::new_bounded(
-        create_limit,
-        std::time::Duration::from_secs(3600),
-        200_000,
-    );
-    if create_limit == 0 {
-        tracing::warn!("GITLAWB_CREATE_RATE_LIMIT=0 — per-IP creation rate limiting disabled");
-    }
+    let create_ip_rate_limiter =
+        build_ip_limiter(create_limit, "GITLAWB_CREATE_RATE_LIMIT", "creation");

     let write_limit = rate_limit_from_env("GITLAWB_WRITE_RATE_LIMIT", 600);
-    let write_rate_limiter = rate_limit::RateLimiter::new_bounded(
-        write_limit,
-        std::time::Duration::from_secs(3600),
-        200_000,
-    );
-    if write_limit == 0 {
-        tracing::warn!("GITLAWB_WRITE_RATE_LIMIT=0 — per-IP write rate limiting disabled");
-    }
+    let write_rate_limiter = build_ip_limiter(write_limit, "GITLAWB_WRITE_RATE_LIMIT", "write");

     let push_limit = rate_limit_from_env("GITLAWB_PUSH_RATE_LIMIT", 600);
-    let push_rate_limiter = rate_limit::RateLimiter::new_bounded(
-        push_limit,
-        std::time::Duration::from_secs(3600),
-        200_000,
-    );
-    if push_limit == 0 {
-        tracing::warn!("GITLAWB_PUSH_RATE_LIMIT=0 — per-IP push rate limiting disabled");
-    }
+    let push_rate_limiter = build_ip_limiter(push_limit, "GITLAWB_PUSH_RATE_LIMIT", "push");
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/main.rs` around lines 292 - 345, Extract the repeated
create, write, and push per-IP limiter initialization into a shared helper that
accepts the environment variable name, default limit, and disabled-warning
context, then returns the bounded RateLimiter. Replace the three inline
rate_limit_from_env/new_bounded/warn sequences with helper calls while
preserving their existing limits, environment overrides, warning behavior, and
bucket duration/capacity.
crates/gitlawb-node/src/git/repo_store.rs (1)

428-446: 🩺 Stability & Availability | 🔵 Trivial | ⚡ Quick win

Make repo lock release drop-safe
RepoLockGuard::release() drops pg_advisory_unlock errors, and the manual guard still depends on every caller remembering .release(). Switching to sqlx::postgres::PgAdvisoryLock/PgAdvisoryLockGuard (or adding a logged, drop-safe fallback) would make leaked locks much harder to miss.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/gitlawb-node/src/git/repo_store.rs` around lines 428 - 446, Update
RepoLockGuard and RepoLockGuard::release so advisory locks are drop-safe: use
sqlx’s PgAdvisoryLock/PgAdvisoryLockGuard where compatible, or add a Drop-based
fallback that attempts cleanup and logs unlock failures. Ensure callers no
longer depend solely on remembering release(), while preserving unlocking on the
same connection.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In @.env.example:
- Around line 143-152: Update the write rate-limiting documentation near
GITLAWB_WRITE_RATE_LIMIT to state that the limiter covers all GraphQL HTTP
requests on the /graphql route, including queries and the playground GET, rather
than describing only GraphQL mutations. Keep the existing rate-limit scope and
configuration details unchanged.

---

Nitpick comments:
In @.env.example:
- Line 19: Move the GITLAWB_METRICS_ADDR entry in .env.example so it appears
before GITLAWB_PORT, preserving the existing key definitions and values.

In `@crates/gitlawb-node/src/git/repo_store.rs`:
- Around line 428-446: Update RepoLockGuard and RepoLockGuard::release so
advisory locks are drop-safe: use sqlx’s PgAdvisoryLock/PgAdvisoryLockGuard
where compatible, or add a Drop-based fallback that attempts cleanup and logs
unlock failures. Ensure callers no longer depend solely on remembering
release(), while preserving unlocking on the same connection.

In `@crates/gitlawb-node/src/main.rs`:
- Around line 292-345: Extract the repeated create, write, and push per-IP
limiter initialization into a shared helper that accepts the environment
variable name, default limit, and disabled-warning context, then returns the
bounded RateLimiter. Replace the three inline
rate_limit_from_env/new_bounded/warn sequences with helper calls while
preserving their existing limits, environment overrides, warning behavior, and
bucket duration/capacity.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 0ed69896-8eea-4a8f-802d-9ceb9db33022

📥 Commits

Reviewing files that changed from the base of the PR and between 27b4a2a and ffdc167.

📒 Files selected for processing (7)
  • .env.example
  • crates/gitlawb-node/src/admin.rs
  • crates/gitlawb-node/src/db/mod.rs
  • crates/gitlawb-node/src/git/repo_store.rs
  • crates/gitlawb-node/src/main.rs
  • crates/gitlawb-node/src/rate_limit.rs
  • crates/gitlawb-node/src/state.rs
🚧 Files skipped from review as they are similar to previous changes (1)
  • crates/gitlawb-node/src/admin.rs

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

  • All required GitHub checks on head da876e8491fe42d276a268b95f32eb1bced1a112 are passing; branch is mergeable against current main (7ae4707c4848a4d8fab75a3edf510e5508d3a8e1).
  • Prior jatmn findings on release-manifest rollback and gitlawb-core dependency-purity gate appear resolved on this head (no .release-please-manifest.json regression in the three-dot diff; workflow gate retained on main).
  • Reconcile merge commit 1459cf2: it dropped durable-upload rollback code that earlier #196 commits on this branch had landed. The P1 item below is not a request to fix novel main-only debt in isolation — it asks you to re-land your own fix and align comments with behavior.

Findings

🟠 P1 — Re-land close/merge durable-upload rollback (lost in merge; comment now overclaims)

📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue), crates/gitlawb-node/src/api/pulls.rs (merge_pr), and the git helpers/tests commit 1d8c281 had added (crates/gitlawb-node/src/git/issues.rs; merge tip restore in git/store.rs for merge_pr).

💥 What fails: A local issue close or branch merge succeeds on disk, then RepoWriteGuard::release hits a definite publish refusal (UploadFailed / Fenced). The handler returns 5xx while the local tree keeps the mutation. RepoStore::acquire fast-path can serve that state until a later refresh.

// close_issue on head (comment promises rollback; implementation does not)
guard.release(close_result.is_ok()).await.into_result()?;

create_issue in the same file already compensates on refusal:

guard.release_compensating(true, move |path| {
    git_issues::delete_issue_ref(&git_bin, path, &issue_id_for_comp, deadline)
})

At merge-base 4fc041a, close_issue and merge_pr already used plain release with no compensating closure — the same observable gap. This PR is not inventing a new class of bug; it claimed to close that class on close/merge, implemented it in 1d8c281, then dropped it in merge 1459cf2 while leaving a comment at close_issue that still describes rollback.

🔎 Root cause: This branch’s merge integration removed compensating release, snapshot helpers, and RED tests, without either restoring the behavior or deleting the misleading comment. merge_pr was not re-synced after the same merge.

📜 Stated contract:

create_issue uses release_compensating on publish failure; this branch’s commits 5a46ef6 / 1d8c281 and the PR description claim improved close/merge behavior when durable storage fails; the in-diff comment at close_issue documents pre-close OID snapshot and rollback.

🏷️ Attribution: Incomplete claimed fix on this branch (merge regression of your own patch). Underlying close/merge behavior without compensation is unchanged pre-existing at merge-base — not a mandate to scope-creep into unrelated main call sites.

📌 In this PR:

  • close_issue handler and the rollback comment block (api/issues.rs)
  • merge_pr handler (api/pulls.rs)
  • re-apply helpers and tests from 1d8c281 (issue_ref_oid, restore_issue_ref, issue_close_failed_upload_rolls_back_to_open, merge analogue)

🔒 Unchanged on main: create_issue compensating pattern, receive-pack path, and Tigris conditional-PUT design — do not redesign upload core.

🔧 Required correction: Cherry-pick/reapply the 1d8c281 shape: capture pre-mutation refs, guard.release_compensating so definite refusal runs rollback while the advisory lock is held, restore helpers and RED tests, and either implement rollback or remove the rollback comment so operator-facing text matches code. Close every listed in-diff row in one pass.

🛠️ Author fix: Re-land the fix this branch already wrote — handlers, helpers, and tests together. Do not patch only the comment or only close_issue.

🚫 Out of scope: Fixing every other write path on main that never used compensation; rebuilding Tigris versioning/conditional-PUT semantics (separate jatmn thread).


🟡 P2 — Purge execute must still delete the object-store archive when disk-path re-validation fails after the DB row is gone

📍 Where: crates/gitlawb-node/src/admin.rs — run_purge_spam execute loop after successful delete_repo_by_id.

💥 What fails: The DB row is deleted and summary.deleted increments, but validated_repo_disk_path then returns Err. The loop continues before delete_archive, leaving the archive in the bucket.

🔎 Root cause: Archive deletion is sequenced after disk-path validation; the error arm exits the iteration without calling repo_store.delete_archive.

📜 Stated contract:

Comments at the execute tail: delete the archive "else it survives and can be downloaded into a later repo created with the same owner/name"; archive_failed exists so survivors are visible.

🏷️ Attribution: PR-introduced (admin.rs is new).

📌 In this PR:

  • run_purge_spam execute loop only

🔧 Required correction: On path-validation failure after a successful row delete, still attempt delete_archive (and count archive_failed on error) before continue, or reorder so archive deletion is not skipped when disk removal is refused.

🛠️ Author fix: Close this path in the same change set as any other purge execute fixes — do not fix archive-only while leaving the symlink recheck gap below unaddressed if you touch ref_count_on_disk.

🚫 Out of scope: Redesigning purge to be transactional across DB+disk+S3.


🟡 P2 — Execute-time emptiness recheck must apply the same symlink fail-closed rule as candidate selection

📍 Where: crates/gitlawb-node/src/admin.rs — ref_count_on_disk (used for under-lock recheck and partition_for_delete) vs local_refs_on_disk.

💥 What fails: Selection rejects a symlink at the repo path (symlink_metadata + warn). Between selection and under-lock recheck, the path can become a symlink to an empty bare repo outside repos_dir. ref_count_on_disk follows the link (HEAD/objects via the target), reports 0 refs, and purge deletes the DB row while path_within may refuse disk removal.

🔎 Root cause: Asymmetric guards: symlink rejection exists only on the selection closure, not on the authoritative recheck function.

📜 Stated contract:

Test symlink_to_outside_empty_bare_fails_closed and module docs: a symlink at the purge target must fail closed and must not be treated as empty-and-purgeable.

🏷️ Attribution: PR-introduced.

📌 In this PR:

  • ref_count_on_disk
  • under-lock recheck at execute (ref_count_on_disk call)
  • add execute-path regression test mirroring the selection symlink test

🔧 Required correction: Share one helper or duplicate the symlink_metadata symlink guard from local_refs_on_disk inside ref_count_on_disk so execute cannot admit symlink targets as empty.

🛠️ Author fix: Fix selection and execute siblings together; add the execute symlink test in the same pass.

🚫 Out of scope: Changing global repo layout or symlink policy outside purge.


🔵 P3 — Keep the exhaustive limiter-sweep regression test aligned with write_rate_limiter

📍 Where: crates/gitlawb-node/src/main.rs — sweep_evicts_expired_keys_from_every_limiter.

💥 What fails: The test’s limiters array still lists ten limiters and omits write_rate_limiter, while AppState::cleanup_rate_limiters sweeps it (H2). Removing write cleanup from production would not fail this test.

📜 Stated contract:

Comment above the test: "Every per-key limiter the router mounts must be swept"; companion cleanup_rate_limiters_reaps_the_write_limiter in state.rs documents H2.

🏷️ Attribution: PR-introduced omission when write_rate_limiter was added.

🔧 Required correction: Add write_rate_limiter to the test seed list and limiters closure (mirror the state-level guard).

🛠️ Author fix: One-line-array update plus assert loop — include in the same commit as any rate-limit touch if you are already editing main.rs tests.

🚫 Out of scope: Changing sweep interval or limiter algorithms.

- try_lock_repo_on routes the acquire through LockProbe so cancellation
  cannot return a lock-owning connection to the pool
- run_pending_migrations runs on the connection holding the advisory
  lock, so a max_connections=1 purge cannot deadlock at startup
- purge-spam canonicalizes repos_dir once; the ./repos default's CurDir
  component made validated_repo_disk_path reject every candidate
- delete_archive fences on the generation verified empty; a PUT landing
  after the check makes the delete refuse instead of erasing it
- refresh and archive delete run under transfer_bound while the repo
  lock is held, so a stalled transfer cannot pin the lock
- init_handle returns the archive upload JoinHandle; create_repo awaits
  it before releasing the creation lock so a purge or retry cannot find
  the row with the PUT still in flight
- ForkCloneGuard is created before the clone and claim_fork_disk_path
  creates the owner dir for the stamp; every clone failure, timeout,
  and cancellation path now cleans only what this attempt owns
- /graphql/ws serves a schema with no mutation root, so the socket can
  no longer execute task mutations outside the write-brake bucket;
  queries and subscriptions still run, and the schema carries the same
  Db, broadcast, and cursor-key data the resolvers consume
- purge-spam skips a repo that still has unsettled receive-pack
  requests or pending ref transitions (checked under the repo lock,
  fail-closed) instead of stranding that work mid-effect
@beardthelion

Copy link
Copy Markdown
Collaborator Author

The coderabbit round is dispositioned on 96ed2b24.

Fixed and resolved:

  • try_lock_repo_on now routes the acquire through LockProbe, so a cancellation racing the advisory-lock query closes the session instead of returning a lock-owning connection to the pool.
  • create_repo awaits the initial archive upload inside the creation lock (init_handle returns the JoinHandle), so a purge or retry cannot find the row with the PUT still in flight.
  • ForkCloneGuard is created before the mirror clone, claim_fork_disk_path creates the owner dir for the stamp, and every failure, timeout, and cancellation path cleans only what the attempt owns.
  • run_purge_spam canonicalizes repos_dir once, fixing the ./repos default that made validated_repo_disk_path reject every default-path candidate.
  • Purge archive refresh and delete run under transfer_bound, and the delete fences on the generation verified empty, so a stalled transfer cannot pin the lock and a racing PUT cannot be erased.
  • run_pending_migrations runs on the connection holding the advisory lock, so a single-connection pool cannot deadlock at startup.
  • /graphql/ws serves a schema with no mutation root, so the socket cannot run writes outside the write-brake bucket.
  • Purge candidacy now excludes repos with unsettled receive-pack requests or pending ref transitions, checked under the lock and fail-closed.

Left open, all one decision: the slug-normalization cluster (canonical-vs-mirror lock keys, full-DID cascade predicates, bounty serialization, sibling-survives policy) plus cache-miss publish fencing. These want a canonical storage/matching form decided once rather than patched predicate-by-predicate in this diff, so they are tracked as a follow-up.

Declined with reasons on the threads: GraphQL reads sharing the write budget (documented tradeoff), trusted-origin gating (no such policy exists), and proxy-port hardening (deployment concern across every limiter).

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

  • GitHub reports mergeable: CONFLICTING / mergeStateStatus: DIRTY on head 96ed2b24ff4fe04679bccde2bd4d7dbf5fca020a. Rebase onto current main, resolve conflicts, and re-run review on the resolved diff.
  • The branch is at least three commits behind origin/main relative to merge-base 4fc041a3995965e199c5dc49d276717b40ab111a; security-sensitive listing/rate-limit work on main may not be present on this head until rebased.
  • CI signal is thin: only CodeRabbit (skipped) on the latest check rollup. Treat merge as blocked until repo-required checks run green on the rebased head.

Findings

🟠 P1 — close_issue documents durable-upload rollback but still uses plain release

📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue).

💥 What fails: The handler closes the issue on disk, then calls guard.release(close_result.is_ok()). If the object-store publish fails, the client gets a 5xx while the local repo still shows the issue closed—same gap create_issue already closes with release_compensating.

// added in this PR — promises rollback; no snapshot or compensator follows
// Snapshot the issue ref's pre-close OID so a failed durable upload can roll
// the close back on local disk, mirroring create_issue. ...

let close_result = git_issues::close_issue(&disk_path, &issue_id);
guard.release(close_result.is_ok()).await.into_result()?;

🔎 Root cause: This diff adds operator-facing text claiming rollback “mirroring create_issue” without wiring the compensating release path that create_issue already uses.

📜 Stated contract:

In-diff comment at close_issue (this PR) requires pre-close snapshot and local rollback on failed durable upload, mirroring create_issue.

create_issue uses release_compensating + git_issues::delete_issue_ref on publish failure.

🏷️ Attribution: Incomplete claimed fix (comment introduced on this head; underlying plain release matches merge-base, but the new comment overclaims).

📌 In this PR:

  • close_issue comment block and release call in api/issues.rs

🔒 Unchanged on main: Tigris publish / RepoWriteGuard core; merge_pr plain release (not in this diff—do not conscript unless you re-land branch rollback work deliberately).

🔧 Required correction: Re-land the compensating pattern (snapshot pre-close ref, release_compensating rollback under the advisory lock, regression test) or remove the rollback comment so code and docs match. Mirror the existing create_issue compensator shape; do not redesign upload core.

🛠️ Author fix: Close the root cause on every listed in-diff row in one pass—comment, release path, and test together. Do not patch only the comment.

🚫 Out of scope: Rewriting merge_pr, receive-pack, or unrelated main call sites unless you explicitly re-import the lost branch rollback commit.


🟡 P2 — Purge execute recheck omits the symlink fail-closed gate used at selection

📍 Where: crates/gitlawb-node/src/admin.rs — local_refs_on_disk vs ref_count_on_disk, execute loop authoritative recheck.

💥 What fails: Selection rejects a symlink at the repo path (fail-closed). Between selection and --execute, a symlink can point at an empty bare repo outside repos_dir. ref_count_on_disk follows the link, reports zero refs, and the row is deleted; path_within may refuse remove_dir_all, producing DB/disk drift.

🔎 Root cause: This PR hardens selection with symlink_metadata but the execute-time authoritative recheck uses ref_count_on_disk, which skips the same symlink gate.

📜 Stated contract:

local_refs_on_disk treats a symlink at the target as non-empty (fail-closed) and documents that a symlink to an outside empty bare repo must never become a candidate.

🏷️ Attribution: PR-introduced asymmetry (new execute recheck path in admin.rs).

📌 In this PR:

  • ref_count_on_disk and execute-loop recheck under the per-repo lock in admin.rs

🔒 Unchanged on main: path_within containment guard; selection tests already cover symlink at selection time.

🔧 Required correction: Share the selection symlink/metadata rules in ref_count_on_disk (or call a single helper) so execute-time emptiness matches selection fail-closed behavior. Add or extend a test that fails if execute recheck follows a symlink.

🛠️ Author fix: Fix the helper and execute path in one pass; do not tighten only dry-run selection.

🚫 Out of scope: Changing EXCLUDED_DIDS / target DID constants; broader purge policy.


🔵 P3 — build_ws_schema rustdoc contradicts the wired schema

📍 Where: crates/gitlawb-node/src/graphql/mod.rs (build_ws_schema).

💥 What fails: New rustdoc says “no query or mutation root exists on this type,” but the function builds QueryRoot + EmptyMutation + SubscriptionRoot. Maintainers can underestimate WS query load (mutations are correctly blocked; queries are not).

🔎 Root cause: In-diff documentation added when splitting WS schema does not match the actual Schema::build call.

📜 Stated contract:

In-diff build_ws_schema doc (this PR) claims the WS schema cannot execute queries or mutations.

🏷️ Attribution: PR-introduced misleading operator/maintainer copy.

📌 In this PR:

  • build_ws_schema module docs in graphql/mod.rs

🔒 Unchanged on main: HTTP /graphql write brake; intentional EmptyMutation on WS.

🔧 Required correction: Rewrite the rustdoc to state mutations are disabled on WS but queries still run via QueryRoot, matching GitlawbWsSchema and server comments about subscriptions-only mutations.

🛠️ Author fix: Update the in-diff doc string in the same pass; do not re-expose MutationRoot on WS unless product intent changes.

🚫 Out of scope: Adding a write brake to /graphql/ws (documented as intentional in .env.example and tests).

…-coverage

# Conflicts:
#	crates/gitlawb-node/src/main.rs
…t ws doc

- close_issue snapshots the pre-close JSON and uses release_compensating,
  so a failed durable upload restores the ref instead of leaving the
  local repo closed while the client saw a 5xx (mirroring create_issue)
- ref_count_on_disk rejects a symlink at the repo path, matching the
  selection-side fail-closed rule so the execute recheck cannot follow
  a link to an empty bare repo outside repos_dir
- build_ws_schema rustdoc now states mutations are disabled via
  EmptyMutation while queries still run on QueryRoot
@beardthelion

Copy link
Copy Markdown
Collaborator Author

All findings are addressed on e9e86a1, on top of the origin/main merge at 082a37f.

Merge readiness: the CONFLICTING state was the main.rs-to-lib.rs split. The branch's entire main.rs delta is re-applied into lib.rs: mod admin, run_admin_command and the admin dispatch, the graphql_ws_schema build and AppState wiring, the build_ip_limiter refactor with the new write_rate_limiter bucket, the cleanup_rate_limiters sweep call sites, and the rate_limit_env_tests module. test_harness.rs builds the ws schema and the write limiter too. Merge state should now be clean.

P1 (close_issue): the compensating pattern is wired, matching create_issue's shape. The handler snapshots the pre-close JSON via get_issue, then calls release_compensating; on a failed durable upload the rollback rewrites the ref to the prior content via create_issue rather than leaving the local repo closed behind a 5xx. Plain release remains only for the close-failed or snapshot-unavailable arms.

P2 (symlink recheck): ref_count_on_disk now refuses a symlink at the repo path (returns non-empty), the same fail-closed rule local_refs_on_disk applies at selection, so the execute-time recheck cannot follow a link to an empty bare repo outside repos_dir.

P3 (ws doc): build_ws_schema rustdoc corrected: mutations are disabled via EmptyMutation while queries still run on QueryRoot (reads that ride the TaskReadBrake), so subscriptions-only refers to the write surface.

Pre-push gate ran green: fmt --check, workspace clippy -D warnings, and the full deny-harness suite (53/53).

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

PR Review — Twigpine/node #196

feat(node): per-IP write-surface rate brake + purge-spam admin tool

Head: e9e86a15d41bddc9b952dbe2cbd2057c22fb0796 · Base: 55ca89bfffa2c81b7f7b6f30cee7743f000197ce

Merge readiness

  • Required check test (beta) failed on run 43114804801: ipfs_pin::tests::discovery_record_elapsed_leaves_neither_row panicked. That test lives in ipfs_pin.rs, which is not in this PR’s 21-file diff; test (stable) passed on the same run. Treat as likely unrelated / flaky on beta until reproduced on a change that touches ipfs_pin, but merge state is BLOCKED until CI is green or maintainers waive beta.
  • Branch is up to date with origin/main (0 behind, 69 ahead). No rebase conflict risk observed.
  • Prior jatmn review threads requested write-brake coverage and purge safety; current head adds broad route tests and admin.rs guards. Remaining gaps are captured in Findings below (not a blanket re-request of older themes).

Findings

🟠 P1 — Purge execute can delete the DB row but leave the object-store archive

📍 Where: crates/gitlawb-node/src/admin.rs — run_purge_spam execute loop, after delete_repo_by_id succeeds.

💥 What fails: On the Ok(n) branch, if validated_repo_disk_path returns Err, the code increments disk_failed and continues before delete_archive runs, even though the DB row was already removed.

🔎 Root cause: Archive deletion is sequenced after disk-path validation; a continue short-circuits the whole tail of the success arm, including object-store cleanup.

📜 Stated contract:

Delete the object-store archive too… else it survives and can be downloaded into a later repo created with the same owner/name.

🏷️ Attribution: PR-introduced (admin.rs is new).

📌 In this PR:

  • run_purge_spam execute path — archive delete skipped when post-delete path validation fails

🔒 Unchanged on main: N/A (new tool).

🔧 Required correction: After a successful delete_repo_by_id, still attempt delete_archive (with the same generation fence) when an object store is configured, even if local disk removal is refused; or reorder so path validation happens before the DB delete. Do not report a clean delete count when the archive was not addressed.

🛠️ Author fix: Close the purge execute lifecycle on every success arm in one pass: DB row, disk dir (when safe), and archive delete/share the same accountability rules. Adjust tests so a post-delete path-validation failure still exercises archive cleanup (or proves intentional skip with an explicit summary counter).

🚫 Out of scope: Redesigning Tigris delete semantics on main.


🟡 P2 — Purge execute summary under-reports unsettled-work skips

📍 Where: crates/gitlawb-node/src/admin.rs — PurgeSummary and final println! after --execute.

💥 What fails:

  • skipped_unsettled is incremented when has_unsettled_repo_work is true, but the closing summary line never prints that counter.
  • When has_unsettled_repo_work returns Err, the repo is skipped fail-closed but no summary field is incremented (unlike skipped_store_error / skipped_locked).

🔎 Root cause: Operator-facing accounting was not kept in sync with the new PurgeSummary fields and the unsettled-work guard.

📜 Stated contract:

skipped_unsettled — Candidates skipped because receive-pack or transition work still has in-flight work on the repo.

🏷️ Attribution: PR-introduced.

📌 In this PR:

  • PurgeSummary struct
  • execute-loop println!
  • has_unsettled_repo_work error arm

🔒 Unchanged on main: N/A.

🔧 Required correction: Include skipped_unsettled in the final execute summary (and dry-run guidance if applicable). Add a dedicated counter (or reuse an existing skip bucket) for unsettled probe failures, with a regression test that forces has_unsettled_repo_work Err and asserts the summary reflects it.

🛠️ Author fix: Align every skip/fail path in the execute loop with a PurgeSummary field and the printed totals in one change.

🚫 Out of scope: Changing unsettled-work SQL semantics.


🟡 P2 — close_issue compensation does not honor git_service_timeout like create_issue

📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue compensating branch (new hunk).

💥 What fails: On successful local close, if durable upload fails, compensation calls git_issues::create_issue(path, …) with no deadline, while the sibling create_issue handler passes git_bin and a git_service_timeout_secs deadline into delete_issue_ref inside release_compensating.

🔎 Root cause: The new close path discards git_bin / deadline (let _ = (git_bin, deadline)) even though the in-diff comment claims mirroring create_issue.

📜 Stated contract:

Snapshot the issue's pre-close JSON so a failed durable upload can roll the close back on local disk, mirroring create_issue.

🏷️ Attribution: PR-introduced.

📌 In this PR:

  • close_issue compensating closure

🔒 Unchanged on main: create_issue compensation pattern (use as reference).

🔧 Required correction: Apply the same timeout-bound git rewrite strategy create_issue uses (or an equivalent bounded compensation) when restoring prior_json after upload failure.

🛠️ Author fix: Make close and create compensation symmetric in one pass; add/extend a test that upload failure on close triggers bounded rollback behavior.

🚫 Out of scope: Rewriting release_compensating for all handlers.


🟡 P2 — create_repo can return 201 while the initial archive upload failed

📍 Where: crates/gitlawb-node/src/api/repos.rs — create_repo (new advisory-lock + create_upload sequence).

💥 What fails: After db.create_repo, the handler awaits the background upload with let _ = upload.await and returns 201 Created regardless of join/upload failure. Comments assert the row, disk, and archive are consistent before the lock is released.

🔎 Root cause: Upload outcome is not propagated to the HTTP response; error paths can also drop the creation lock before the upload task finishes (success path only awaits upload).

📜 Stated contract:

Keep the initial archive upload inside the creation lock… Row + on-disk dir now both exist consistently; the race window is closed.

🏷️ Attribution: PR-introduced.

📌 In this PR:

  • create_repo success and error paths around create_upload / repo_lock.release()

🔒 Unchanged on main: Prior create flow without this lock/upload ordering.

🔧 Required correction: Treat a failed create-only upload like other archive failures relevant to creation (fail the request or compensating-delete row+disk), and keep the advisory lock until the upload task completes on all exit paths that inserted a row, matching the stated race comment.

🛠️ Author fix: Close create/upload/lock lifecycle on every exit path in one pass; add regression coverage for upload failure during create under Tigris.

🚫 Out of scope: Changing fork/upload semantics outside create_repo.


🔵 P3 — purge-spam cannot take GITLAWB_TIGRIS_BUCKET from CLI after the subcommand

📍 Where: crates/gitlawb-node/src/config.rs (tigris_bucket not global = true); lib.rs run_admin_command.

💥 What fails: gitlawb-node purge-spam --tigris-bucket … is rejected by clap; only repos_dir and database_url are global. Tigris-backed authoritative purge requires env or flags before purge-spam.

🔎 Root cause: Admin wiring reads config.tigris_bucket, but the CLI surface for purge does not expose it post-subcommand.

📜 Stated contract:

Build the object store so purge is authoritative against remote state… When no bucket is configured this stays None and purge is local-only.

🏷️ Attribution: PR-introduced admin entrypoint.

📌 In this PR:

  • PurgeSpam subcommand
  • run_admin_command object-store setup

🔒 Unchanged on main: Daemon flag layout.

🔧 Required correction: Mark tigris_bucket (and any other store fields purge needs) global = true, or document in purge-spam --help / operator runbooks that bucket config must come from env or pre-subcommand flags.

🛠️ Author fix: Pick one operator path and make help, .env.example, and clap agree in one pass.

🚫 Out of scope: New object-store providers.

- purge-spam execute: the object-store archive delete now runs on every
  successful row delete; a refused or failed local dir removal no longer
  strands the archive (P1)
- purge-spam summary: skipped_unsettled is now printed, and a dedicated
  skipped_unsettled_probe counter records fail-closed skips when the
  unsettled-work probe itself errors (P2)
- close_issue: rollback compensation uses create_issue_bounded, the same
  deadline-bounded git strategy as create_issue's compensator, so a hung
  git cannot stall the write guard past git_service_timeout (P2)
- create_repo: the initial archive upload's outcome now gates the 201; a
  failed or fenced upload rolls back the row and disk dir and surfaces
  an error instead of reporting success (P2)
- purge-spam: tigris_bucket is clap-global so --tigris-bucket parses
  after the subcommand (P3)
@beardthelion

Copy link
Copy Markdown
Collaborator Author

All five findings are addressed on 1187dca.

P1 (archive orphan): the execute loop no longer continues past delete_archive on a disk-path validation failure. The Ok(n) arm now treats disk removal as best-effort and always runs the archive delete (same generation fence), so a refused or failed local removal cannot strand the object-store copy of a row that was just deleted.

P2 (unsettled accounting): skipped_unsettled is now in the final summary line, and a dedicated skipped_unsettled_probe counter records the fail-closed skip when has_unsettled_repo_work itself errors, so an operator can distinguish work in flight from could-not-check.

P2 (bounded close rollback): git/issues.rs gained create_issue_bounded, which runs the same hash-object plus update-ref write through run_bounded_git with the caller deadline and git_bin. close_issue compensator calls it, matching create_issue deadline strategy; the discard of git_bin and deadline is gone.

P2 (create_repo upload): init_handle upload task now returns a Result, and create_repo awaits it. An upload failure, a precondition-lost fence (the archive key then belongs to another writer), or a task panic rolls back the create: the row is deleted via delete_repo_by_id, the disk dir removed, the lock released, and the client sees an error instead of 201.

P3 (tigris flag): tigris_bucket is now clap-global, so gitlawb-node purge-spam --tigris-bucket parses post-subcommand. Pre-subcommand and env paths still work.

Pre-push gate ran green: fmt --check, workspace clippy -D warnings, deny_harness 53/53. The deferred slug-normalization and cache-miss threads remain open as the tracked follow-up set.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

  • All required checks are green on head 1187dcab79c93fb38188c50138d30db65f00e265 (stable + beta tests, fmt/clippy, build).
  • Branch is 0 commits behind origin/main (merge-base 55ca89bfffa2c81b7f7b6f30cee7743f000197ce); mergeable.
  • GitHub review decision is still CHANGES_REQUESTED from earlier heads; several prior jatmn items (close_issue compensating release, purge archive/symlink siblings) are fixed on this head — see findings below for what remains.

Findings

🟠 P1 — Re-land merge durable-upload rollback claimed in release notes (close path fixed; merge path not)

📍 Where: crates/gitlawb-node/src/api/pulls.rs (merge_pr).

💥 What fails: store::merge_branch mutates the bare repo, then guard.release(merge_result.is_ok()) publishes to durable storage. On definite publish refusal the handler returns 5xx, but there is no git rollback of the merge commit (unlike close_issue, which now uses release_compensating with a pre-close snapshot in api/issues.rs).

guard.release(merge_result.is_ok()).await.into_result()?;
let merge_sha = merge_result.map_err(|e| AppError::Git(e.to_string()))?;
state.db.merge_pr(&pr.id, &merger_did).await?;

🔎 Root cause: This PR advertises improved close/merge behavior on storage failure and implements compensation for close, but merge_pr still uses plain release with no compensating closure.

📜 Stated contract:

CodeRabbit release notes on this PR: "Improved pull-request merge handling when durable storage operations fail."

🏷️ Attribution: Incomplete claimed fix on this branch (close fixed; merge not). Underlying merge-without-compensation also exists at merge-base — not a mandate to refactor all main writers.

📌 In this PR:

  • merge_pr handler (pulls.rs)
  • any merge rollback helpers/tests this branch previously added for merge_pr but did not keep on head

🔒 Unchanged on main: RepoWriteGuard publish/quarantine core, receive_pack path, Tigris conditional PUT semantics.

🔧 Required correction: Mirror the close_issue / create_issue pattern for merge_pr: capture pre-merge refs (or merge parent), use release_compensating so definite publish refusal rewinds the local merge while the advisory lock is held, and add a RED test analogous to create_issue_rolls_back_local_ref_when_publish_refuses.

🛠️ Author fix: Land merge compensation, helpers, and tests together in one pass. Do not patch only release notes or only comments.

🚫 Out of scope: Rebuilding every other write handler on main that never compensated.


🟡 P2 — create_repo upload rollback overclaims when the DB row delete fails

📍 Where: crates/gitlawb-node/src/api/repos.rs (create_repo, initial archive upload inside the creation lock).

💥 What fails: When the empty-repo upload fails, the handler calls delete_repo_by_id, then remove_dir_all, then returns an error stating the create was rolled back. If delete_repo_by_id errors (warn-only), execution still removes the on-disk directory and returns the rollback wording.

🔎 Root cause: Rollback success is not gated on both row delete and dir removal succeeding; operator-visible text overstates recovery.

📜 Stated contract:

In-diff comment above the upload block: a failed upload must "roll the create back (row + dir)" before surfacing the error instead of reporting success.

🏷️ Attribution: PR-introduced (new upload-in-lock + rollback block on this branch).

📌 In this PR:

  • create_repo upload failure path only

🔒 Unchanged on main: Proof consume ordering, iCaptcha core, unrelated handlers.

🔧 Required correction: If delete_repo_by_id fails, do not claim full rollback — return an error that states partial state (orphan row and/or orphan dir) and/or retry row delete before removing disk; add a regression test where mocked delete_repo_by_id fails.

🛠️ Author fix: Close the handler message and control flow together; do not change only the user-facing string.

🚫 Out of scope: Redesigning repo creation or Tigris upload pipeline.


🟡 P2 — Purge execute: post-commit window allows mirror ingest before remove_dir_all

📍 Where: crates/gitlawb-node/src/admin.rs (run_purge_spam execute loop); interacts with crates/gitlawb-node/src/db/mod.rs (delete_repo_by_id slug xact lock) and upsert_mirror_repo.

💥 What fails: Purge holds try_lock_repo (SHA-based session advisory lock) across delete + disk removal. delete_repo_by_id takes the slug hashtextextended transaction lock only until commit. After the row delete commits, a concurrent upsert_mirror_repo can insert a new repos row for the same normalized slug while purge still runs remove_dir_all on that owner/name path — wiping disk under a live row.

🔎 Root cause: Slug-level serialization ends at transaction commit; on-disk deletion is not serialized against mirror ingest the way the in-transaction sibling check is.

📜 Stated contract:

delete_repo_by_id comment: slug xact lock serializes purge against concurrent upsert_mirror_repo; purge module goal that DB row and on-disk dir stay consistent.

🏷️ Attribution: PR-introduced (admin.rs new; expanded cascade in this diff). DB test purge_serializes_with_mirror_ingest_via_advisory_lock holds a session slug lock for the whole scenario — live purge does not hold that lock across disk delete.

📌 In this PR:

  • run_purge_spam execute loop (after successful delete_repo_by_id, before / during disk removal)
  • add execute-path regression mirroring the DB race test with the real purge lock sequence

🔒 Unchanged on main: Global mirror ingest API; do not remove slug lock from upsert_mirror_repo.

🔧 Required correction: Hold slug advisory serialization from before delete through disk removal, or re-check sibling_exists / row absence immediately before remove_dir_all and abort disk delete if a new row landed.

🛠️ Author fix: Close purge execute + test in one pass.

🚫 Out of scope: Transactional purge across DB+disk+S3 in one atomic step.


🔵 P3 — Sweep integration test still omits write_rate_limiter

📍 Where: crates/gitlawb-node/src/lib.rs — sweep_evicts_expired_keys_from_every_limiter.

💥 What fails: The test seeds and asserts cleanup for ten limiters but not write_rate_limiter, even though AppState::cleanup_rate_limiters sweeps it (H2; guarded separately in state.rs).

🔎 Root cause: New limiter added in this PR without extending the cross-limiter regression array.

📜 Stated contract:

Comment above the test: "Every per-key limiter the router mounts must be swept."

🏷️ Attribution: PR-introduced omission.

📌 In this PR:

  • sweep_evicts_expired_keys_from_every_limiter limiter list only

🔒 Unchanged on main: Sweep interval algorithms.

🔧 Required correction: Add state.write_rate_limiter to the seed loop and limiters closure (same pattern as cleanup_rate_limiters_reaps_the_write_limiter).

🛠️ Author fix: One-line array update in the same commit if touching limiter tests.

🚫 Out of scope: Changing production cleanup wiring (already correct in state.rs).

- merge_pr snapshots the pre-merge target tip and uses
  release_compensating to rewind it via bounded update-ref on a
  refused publish, matching the close_issue pattern; regression proves
  the ref returns to its prior OID after a refused upload (P1)
- create_repo rollback names residual state instead of claiming a full
  rollback when the row delete fails or dir removal fails (P2)
- purge execute re-checks row absence via get_repo_by_id immediately
  before remove_dir_all and refuses the disk delete when a row
  reappeared after commit, closing the post-commit mirror-ingest
  window (P2)
- sweep_evicts_expired_keys_from_every_limiter now seeds and overrides
  the write_rate_limiter TTL (P3)
@beardthelion

Copy link
Copy Markdown
Collaborator Author

All four findings are addressed on 261f7ab.

P1 (merge rollback): merge_pr now snapshots the target ref pre-merge OID via read_ref_bounded and uses release_compensating, whose closure rewinds refs/heads/ through run_bounded_git inside git_service_timeout, the same shape as close_issue. merge_pr_rewinds_local_merge_when_publish_refuses asserts rev-parse returns to the pre-merge tip after a refused publish.

P2 (create_repo overclaim): the rollback now tracks row_deleted and dir_removed independently, and the returned error names the residual state (the repo row remains / the on-disk directory remains / both) with operator cleanup required, instead of claiming a full rollback when either step failed.

P2 (purge post-commit window): the execute loop re-checks db.get_repo_by_id immediately before remove_dir_all and refuses the disk delete when a repo row reappeared after commit, counting disk_failed; a probe error fails closed.

P3 (sweep omission): sweep_evicts_expired_keys_from_every_limiter now lists write_rate_limiter in the limiters closure and gives it the same RateLimiter::new(10, window) TTL override as the others; the omission was in both spots, which is why the first pass of the fix went red.

Pre-push gate ran green: fmt --check, workspace clippy -D warnings, deny_harness 53/53. The deferred slug-normalization and cache-miss threads remain open as the tracked follow-up set.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

Required checks are red on this head. GitHub Actions run 43142788397 reports test (stable) and test (beta) failed on cargo test, while fmt + clippy, build --release, MSRV, and Docker smoke passed. The live target branch main at merge-base 55ca89b is green on the same workflow class. The PR verification section claims cargo test --workspace is green — that does not match the required checks on 261f7ab3. Reconcile the failing test names from CI logs before merge (this is merge hygiene on this head, not a mandate to fix unrelated main CI debt).

Base is current. Merge-base equals live main (55ca89b); no stale-base rebase is required for metadata drift.


🟠 P1 — Post-delete purge must probe slug liveness before disk or archive destruction

📍 Where: crates/gitlawb-node/src/admin.rs (run_purge_spam execute loop, post-delete_repo_by_id disk removal and archive delete).

💥 What fails: After the canonical row is deleted, the pre-remove_dir_all guard calls db.get_repo_by_id(&c.id) for the purged primary key. A concurrent upsert_mirror_repo can insert a new row for the same owner/name slug with id {owner_short}/{name} (db/mod.rs) while purge still holds the session try_lock_repo lock — upsert_mirror_repo does not take that session lock, only the transaction-scoped lock_repo_slug that ends when delete_repo_by_id commits.

The inline comment names this mirror-ingest window, but the probe cannot see the new row:

// concurrent mirror ingest can insert a new row for this slug before remove_dir_all runs.
let row_gone = match db.get_repo_by_id(&c.id).await {
    Ok(None) => true,
    // ...
};

When row_gone is wrongly true, purge removes the on-disk bare repo. Archive delete still runs unconditionally afterward (delete_archive is not gated on slug absence), so object-store bytes can be removed while a live mirror repos row remains — DB / disk / archive drift and broken serve for that slug.

🔎 Root cause: Post-delete destructive steps use primary-key absence instead of slug-level liveness. The PR’s own comment describes slug-level re-check; the implementation checks only the deleted id.

📜 Stated contract:

Re-check immediately before the destructive op and abort the disk delete when a fresh row landed rather than wipe disk under a live row.

🏷️ Attribution: PR-introduced. There is no crates/gitlawb-node/src/admin.rs at merge-base 55ca89b; this execute path and guard are new in this diff.

📌 In this PR:

  • run_purge_spam post-delete disk guard (get_repo_by_id)
  • unconditional delete_archive after row delete
  • operator dry-run / execute summary that can report disk+archive removal under a surviving slug row

🔒 Unchanged on main: In-transaction lock_repo_slug serialization between delete_repo_by_id and upsert_mirror_repo (existing DB test purge_serializes_with_mirror_ingest_via_advisory_lock). This finding is only the post-commit window after that xact lock is released.

🔧 Required correction: Before remove_dir_all and before delete_archive, treat the slug as live when db.get_repo(&c.owner_did, &c.name) (or equivalent owner-key-normalized lookup) returns any row. Skip both destructive steps when a row exists; count separately in PurgeSummary. Add a regression test racing upsert_mirror_repo after delete_repo_by_id commits but before disk removal (mirror id owner/name).

🛠️ Author fix: Close slug-level liveness on both disk and archive paths in one pass; do not patch only the get_repo_by_id call while leaving archive delete ungated.

🚫 Out of scope: Reworking peer-sync ingest to take session repo locks; changing mirror id format; fixing pre-existing ingest paths on unchanged main.


🟡 P2 — Issue write HTTP brake must not run before the INV-12 read gate

📍 Where: crates/gitlawb-node/src/server.rs (issue_write_routes + write_braked); crates/gitlawb-node/src/api/issues.rs (close_issue, create_issue_comment).

💥 What fails: issue_write_routes is wrapped with write_braked, so rate_limit_by_ip on the write bucket is the outermost middleware on:

  • POST /api/v1/repos/{owner}/{repo}/issues/{id}/close
  • POST /api/v1/repos/{owner}/{repo}/issues/{id}/comments

An exhausted write bucket returns 429 before authorize_repo_read runs in the handler. For callers probing private or missing repos, that breaks the documented status ordering: 429 can appear without the read gate having returned the opaque 404 used for hidden repos.

🔎 Root cause: This PR applied the shared HTTP write brake to the issue write router. The INV-12 ordering comment in close_issue (and the read-gated comment flow) predates this PR unchanged at 55ca89b; merge-base issue_write_routes had no .write_braked. The new outer middleware runs before the handler; the in-handler close_issue_rate_limiter still honors post-gate ordering but does not undo the outer brake.

📜 Stated contract:

Rate limiting runs AFTER this gate so a 429 cannot distinguish a hidden repo from a missing one (INV-12 read-denial status contract).

🏷️ Attribution: PR-activated. Merge-base issue_write_routes ended at add_auth_layers with no write_braked; head adds .write_braked(&write_ip_limiter).

📌 In this PR:

  • server.rs issue_write_routes.write_braked
  • close_issue (read-gated close + inner close_issue_rate_limiter)
  • create_issue_comment (read-gated comment write)
  • new integration tests that assert write brake on issue comment without INV-12 ordering

🔒 Unchanged on main: Other write groups (merges, stars, hooks, etc.) that are not read-gated before mutation may keep outer write_braked; do not remove their middleware or extend INV-12 refactors to those routes in this PR.

🔧 Required correction: Debit the write bucket for issue close and issue comment inside each handler immediately after authorize_repo_read (same placement as close_issue_rate_limiter), or split issue close/comment onto a router without outer write_braked while still debiting write_rate_limiter post-gate. Add regression tests: under an exhausted write bucket, a caller denied read access still receives the read-gate status (404/403), not 429, for both routes.

🛠️ Author fix: Fix ordering on both issue write routes together; do not move only close_issue while leaving comment on the outer brake.

🚫 Out of scope: Changing INV-12 behavior on unrelated read routes; redesigning global write-brake routing for the whole server; pre-existing GITLAWB_CLOSE_ISSUE_RATE_LIMIT env parsing (unchanged at merge-base).

…he read gate (#196)

Three items from review round 6:

- merge_pr rollback test seeded commits with git commit-tree under no
  configured identity, which fails on CI runners. The helper now passes
  -c user.name/-c user.email per invocation.
- run_purge_spam's post-delete guard probed the deleted primary key, but
  a concurrent upsert_mirror_repo lands a NEW row for the same slug with
  a mirror id, and the archive delete ran unconditionally. The probe now
  reads slug liveness via get_repo(owner, name) and gates BOTH the disk
  and archive deletes, counted as skipped_slug_live.
- issue_write_routes carried the outer per-IP write brake, so an
  exhausted bucket answered 429 before authorize_repo_read could return
  the opaque 404 a hidden repo owes the caller. The brake is now debited
  inside close_issue and create_issue_comment, immediately after the
  read gate; the route group carries no outer brake.
@beardthelion

Copy link
Copy Markdown
Collaborator Author

All three items fixed in 662b9a3a.

Merge readiness: test (stable)/test (beta) red was merge_pr_rewinds_local_merge_when_publish_refuses, which seeded commits with git commit-tree under no configured author identity and fails on CI runners. The test helper now passes -c user.name/-c user.email per invocation. The beta-only reconciliation::sweep_restores_tigris_cache_miss_and_skips_missing failure is in a file this diff does not touch (a sweep-scheduling assertion); flagging it as pre-existing, not reconciled here.

P1, post-delete slug liveness: the guard now probes get_repo(owner, name), so a concurrent upsert_mirror_repo landing a new row for the same owner/name (mirror id, not the deleted primary key) reads the slug as live. The probe gates BOTH remove_dir_all and delete_archive (the archive delete is no longer unconditional), and a probe error fails closed. New skipped_slug_live bucket in PurgeSummary and the summary line. The post-delete block is factored into purge_deleted_repo_storage, so the post-commit window is drivable directly: the regression test deletes the canonical row, upserts a mirror for the same slug, and asserts both destructive steps refuse with the dir intact.

P2, issue write brake ordering: issue_write_routes no longer carries the outer write_braked. close_issue debits write_rate_limiter next to its existing post-gate close_issue_rate_limiter debit, and create_issue_comment debits write_rate_limiter post-gate with new headers/PeerAddr extractors. The regression test empties the shared write bucket, then drives a read-denied caller through both routes and asserts the read-gate RepoNotFound, not 429.

Verified locally: cargo check, clippy --workspace --all-targets -D warnings, fmt, 53/53 deny-harness, and all three new regression tests green.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

  • Required GitHub checks test (stable) and test (beta) failed on head 662b9a3a31cbc973d4a4f728fcb5ccfe74f3b3aa (workflow run 43164147039). fmt, clippy, build, MSRV, and other gates are green.
  • PR-owned CI blocker: Finding 1 below (issue_comment_is_rate_limited_by_ip) fails locally and is in this diff; fixing or removing it is required for a green cargo test on this branch.
  • Pre-existing CI friction (not in ## Findings): test_support::tests::merge_pr_fails_closed_when_target_is_only_a_tag and merge_pr_advances_local_branch_shadowed_by_same_named_tag call git tag release without -m or -F; on dumb CI terminals that fails with “EDITOR unset.” The same invocation is present at merge-base 55ca89bf — not introduced by this feature diff. It can still block required checks on this head until those helpers pass -m (or CI sets a tag message). That is merge hygiene, not a request to redesign merge semantics in this PR.
  • Out of this diff: If test (beta) also reports reconciliation::sweep_restores_tigris_cache_miss_and_skips_missing, that file is unchanged on this branch; treat as upstream flake or main-track follow-up, not a #196 code finding.
  • Branch is mergeable against current origin/main (55ca89bfffa2c81b7f7b6f30cee7743f000197ce, 0 behind). No conflict markers in the prepared checkout.
  • Prior review threads on advisory-lock pinning, purge/mirror serialization, merge/close compensating upload refusal (merge path), archive delete after row delete, symlink recheck, and limiter sweep are addressed on this head per code inspection and your recent comments; not re-opened except where evidence below still fails.
  • Drift check (withdrawn from draft): Outer write_braked on star/merge routes does not violate INV-12 the way issue routes do (exhausted bucket yields the same 429 before the handler for missing and real slugs). Invalid GITLAWB_*_RATE_LIMIT parse → default with warn is intentional. Purge/mirror xact slug lock and related tests match your stated fixes.

🟠 P1 — Remove or rewrite the issue-comment router throttle test so it matches INV-12 wiring

📍 Where: crates/gitlawb-node/src/api/repos.rs — issue_comment_is_rate_limited_by_ip

💥 What fails: The test exhausts write_rate_limiter, POSTs to /api/v1/repos/someowner/somerepo/issues/1/comments without a signature, and expects 429. The router returns 401 because issue write routes deliberately omit the outer write_braked middleware; throttling runs inside create_issue_comment only after authorize_repo_read and authentication.

assertion `left == right` failed: issue-write routes must be IP-throttled by the write brake
  left: 401
 right: 429

🔎 Root cause: The test was added as a “representative REST write group” sibling to star/label routes that use write_braked, but issue comments follow the in-handler brake pattern documented for INV-12.

📜 Stated contract:

Issue write routes: an outer per-IP brake would answer 429 before authorize_repo_read could return the opaque 404 a hidden repo owes the caller (INV-12).

🏷️ Attribution: PR-introduced (this test is new in the diff).

📌 In this PR:

  • crates/gitlawb-node/src/api/repos.rs — delete issue_comment_is_rate_limited_by_ip, or replace it with a test that signs/auths and asserts 429 from the handler path (mirror issues.rs an_exhausted_write_bucket_cannot_preempt_the_read_gate / existing create_issue_comment rate tests)

🔒 Unchanged on main: server.rs issue route group without outer brake; do not “fix” by attaching write_braked to issue routes.

🔧 Required correction: Drop the misleading router-only test or rewrite it to exercise in-handler debit with proper auth and read-gate ordering.

🛠️ Author fix: One pass on repos.rs tests only; keep coverage in issues.rs as the authority for issue-write throttle ordering.

🚫 Out of scope: Moving issue routes to middleware-level throttling.


🟡 P2 — Close-issue upload rollback must rewrite the resolved issue ref, not the URL prefix

📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue compensating release_compensating closure

💥 What fails: When the client closes via an 8-char prefix, git_issues::close_issue resolves to the full UUID and updates refs/gitlawb/issues/{full_id}. If durable upload then refuses, the compensator calls create_issue_bounded with the path issue_id (the prefix), creating/updating refs/gitlawb/issues/{prefix} instead of the canonical ref. The client sees 5xx while the full ref stays closed.

🔎 Root cause: Compensating rollback was added on this branch (merge-base used plain release on close with no upload compensation). The compensator copies the path segment into the git ref writer without using the same resolved id close_issue used for the mutation.

📜 Stated contract:

Handler comment: on failed upload, “rewrites the ref back to the pre-close content instead of leaving the local repo closed while the client saw 5xx.”

git/issues.rs test_close_issue_via_prefix: close-by-prefix mutates the resolved full ref.

🏷️ Attribution: PR-introduced (compensating close on close is new on this branch).

📌 In this PR:

  • crates/gitlawb-node/src/api/issues.rs — compensator id passed to create_issue_bounded
  • Add a regression test: close via prefix + refused publish → local ref content returns to pre-close on the full id

🔒 Unchanged on main: Tigris upload core; create_issue compensator pattern.

🔧 Required correction: Derive the resolved full issue id (from close_result or resolve_issue_id) and pass that into create_issue_bounded in the compensator.

🛠️ Author fix: Handler + one load-bearing test in the same commit; do not change unrelated write paths.

🚫 Out of scope: Prefix resolution behavior in git/issues.rs itself; compensating every other write path on main that never had rollback at merge-base.

…test, tag fixtures get -m (#196)

- close_issue's compensating rollback now rewrites the resolved full issue
  id; a close-by-prefix refused at publish left the full ref closed while
  the client saw 5xx (P2, regression test included)
- issue_comment_is_rate_limited_by_ip asserted an outer 429 the issue
  write routes deliberately lack (P1); issues.rs owns the ordering
- git tag fixtures pass -m so dumb CI terminals do not hit "EDITOR unset"
@beardthelion

Copy link
Copy Markdown
Collaborator Author

All three items are on 43d4f59f.

  • P1 (brake test): issue_comment_is_rate_limited_by_ip is deleted. It asserted a router-level 429 the issue write routes deliberately do not have; issues.rs stays the authority for the in-handler brake ordering (an_exhausted_write_bucket_cannot_preempt_the_read_gate and the shed tests).
  • P2 (close rollback): close_issue now resolves the path id via git_issues::resolve_issue_id before the close and binds the resolved full id into the release_compensating arm, so a refused publish rewrites the same ref the mutation touched. close_via_prefix_restores_the_full_ref_when_publish_refuses drives close-by-prefix plus a definite refusal and asserts the pre-close content is restored on the full-id ref. One fixture note: a definite refusal invalidates the live tree before the compensator runs, so the test pins the tree in place to observe the rollback; the doc comment says so.
  • git tag hygiene: both fixtures in test_support.rs pass -m, so the tests no longer need an editor on a dumb terminal.

On reconciliation::sweep_restores_tigris_cache_miss_and_skips_missing: agreed, that file is untouched here, treated as out-of-diff flake.

The earlier coderabbit deferrals are now tracked in #583 and those threads are resolved.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I found issues that need to be addressed before this is ready.

Merge readiness

test (stable) failed on the latest workflow run (job log); test (beta) passed. Treat the stable failure as a merge blocker until the failing case is confirmed PR-owned or the run is green on head.

GitHub still reports mergeStateStatus: BLOCKED. fmt/clippy, build --release, MSRV, cargo audit, and Rust CodeQL passed on that run.

The PR head (43d4f59f) matches the fetched origin/main merge-base (55ca89bf) — no stale-branch conflict surfaced in this checkout.

Findings

🟡 P2 — Align in-diff operator copy with every surface that debits GITLAWB_WRITE_RATE_LIMIT

📍 Where: .env.example (new write-rate section), crates/gitlawb-node/src/lib.rs comment above write_rate_limiter, crates/gitlawb-node/src/state.rs write_rate_limiter field rustdoc, and the PR description consumer list (same omissions as .env.example).

💥 What fails: An operator sets GITLAWB_WRITE_RATE_LIMIT from the new template (or reads the PR body) and treats the bullet list as the full per-IP aggregate. On head, these also debit the shared write bucket but are missing or understated in that copy:

  • POST …/issues/{id}/close — handler-level debit of write_rate_limiter new in this PR (merge-base only used close_issue_rate_limiter), plus the separate GITLAWB_CLOSE_ISSUE_RATE_LIMIT
  • POST …/issues/{id}/comments — handler-level write_rate_limiter new in this PR
  • Webhook POST / DELETE on write_routes — routes existed at merge-base but were unbraked; this PR puts them behind write_braked
  • GET …/visibility on the same write_braked group — intentional in code (harmless read debiting write bucket) but not called out in the new env text
  • state.rs rustdoc still says GraphQL MutationRoot only, while HTTP /graphql debits queries and playground GET as well (.env.example documents HTTP; field doc does not)

This is operator-contract drift on surfaces this PR newly attached to the bucket, not a request to fix pre-existing unbraked behavior on main.

🔎 Root cause: The write brake landed with a new env section and rustdocs that repeat an incomplete enumeration; the implementation and server.rs wiring are broader than the listed consumers.

📜 Stated contract:

Max non-creation write requests per client IP per hour: issue/PR comments, labels, stars, merges, protect/unprotect, replicas, visibility, tasks, bounties, profile, and all GraphQL HTTP requests.

— new .env.example section on this branch (presents itself as the consumer inventory for GITLAWB_WRITE_RATE_LIMIT)

🏷️ Attribution: PR-introduced copy drift (new env block; new write_rate_limiter debits on issue paths; webhooks newly behind write_braked). Not a pre-existing main doc bug — merge-base had no GITLAWB_WRITE_RATE_LIMIT section and did not debit the write bucket on close_issue.

📌 In this PR:

  • .env.example write-rate section
  • lib.rs comment above build_ip_limiter("GITLAWB_WRITE_RATE_LIMIT", …)
  • state.rs write_rate_limiter field documentation
  • PR description “What” bullet list (if still posted as operator guidance)
  • Short note: issue close/comment use handler debits (INV-12), not outer write_braked

🔒 Unchanged on main: Bucket limits, creation/push/peer buckets, and route behavior outside this diff.

🔧 Required correction: Rewrite every in-diff operator surface to the same complete consumer list for head: add issue close, issue comments, webhooks, dual-bucket note for close, visibility GET on the write group, and HTTP GraphQL (queries + playground GET, not only mutations). Align state.rs rustdoc with .env.example on GraphQL HTTP vs WebSocket.

🛠️ Author fix: Update all listed surfaces in one pass so no sibling comment or env block still advertises the short list.

🚫 Out of scope: Retuning defaults, new env vars, or moving issue routes to router middleware (INV-12).


🟡 P2 — close_issue must not consume write_rate_limiter when only the close cap rejects

📍 Where: crates/gitlawb-node/src/api/issues.rs — close_issue; crates/gitlawb-node/src/rate_limit.rs — check / is_throttled.

💥 What fails: After the read gate:

if !state.write_rate_limiter.check(&key).await
    || !state.close_issue_rate_limiter.check(&key).await

check records a timestamp on allow. If the write bucket has capacity but GITLAWB_CLOSE_ISSUE_RATE_LIMIT is exhausted, the handler returns 429 after consuming one write slot. The IP loses shared write quota (stars, comments, GraphQL HTTP, etc.) without completing a close.

At merge-base, close_issue only called close_issue_rate_limiter.check — this cross-bucket bleed does not exist on main.

🔎 Root cause: This PR added write_rate_limiter to close_issue but chains two consuming checks with ||, so the first successful check always records even when the second rejects.

📜 Stated contract:

push traffic must keep its own bucket after close_issue is exhausted

— close_issue_rate_limit_does_not_drain_push_bucket in issues.rs (existing test that encodes cross-bucket isolation for close vs push; same product rule applies to close vs the shared write bucket)

📜 Stated contract (mechanism):

Unlike [check], it records nothing and never inserts a new key … WITHOUT perturbing the per-unit budget the consuming check maintains.

— RateLimiter::is_throttled in rate_limit.rs

🏷️ Attribution: PR-introduced (write_rate_limiter.check added in this diff only). Pre-existing close-only limiting on main is not the defect; the new write debit is.

📌 In this PR:

  • close_issue handler debit logic (after authorize_repo_read, before auth work)
  • New regression test: exhaust close bucket while write bucket has capacity; assert write counter/timestamp unchanged (mirror close_issue_rate_limit_does_not_drain_push_bucket)

🔒 Unchanged on main: INV-12 ordering (read gate before 429); issue routes must stay off outer write_braked.

🔧 Required correction: Use non-consuming is_throttled on both buckets to decide rejection; only call consuming check on both when the request will proceed past the rate gates (or equivalent logic with no partial debit). Add the regression test above.

🛠️ Author fix: Fix dual-bucket behavior and test in the same commit; do not merge the two limiters or add router middleware on issue routes.

🚫 Out of scope: Changing GITLAWB_CLOSE_ISSUE_RATE_LIMIT parsing (unchanged vs base), other write routes, or framework-wide rate-limit redesign.

… complete write-brake operator copy (#196)

close_issue chained two consuming limiter checks, so a close refused by its
own cap still debited the shared write bucket. Probe both with is_throttled
first and consume only when both have room. Rewrite the write-brake consumer
list in .env.example, lib.rs and state.rs to cover every surface on the bucket.
@beardthelion

Copy link
Copy Markdown
Collaborator Author

Both P2s are addressed on 4b483a3.

close_issue must not consume write_rate_limiter when only the close cap rejects

close_issue now probes both buckets with is_throttled first and only runs the consuming check on each once both have room, so a close refused by either cap spends nothing from the other. Two regression tests in api/issues.rs: close_issue_refused_by_close_cap_does_not_debit_write_bucket failed before the change (the write slot was gone after the 429) and passes now, and close_issue_refused_by_write_cap_does_not_debit_close_bucket covers the reverse direction.

Align in-diff operator copy with every surface that debits GITLAWB_WRITE_RATE_LIMIT

The .env.example section, the lib.rs comment and the state.rs field rustdoc now list the same consumers: issue close and comments, webhook create/delete, the visibility GET on that group, and every HTTP /graphql request (WebSocket excluded). They also say that issue close and comment debit in the handler after the read gate, and that close also draws on GITLAWB_CLOSE_ISSUE_RATE_LIMIT.

test (stable) failed

The failing case on the previous head was reconciliation::tests::sweep_persists_last_attempted_not_planned_tail_on_mid_batch_narrow. This branch does not touch reconciliation.rs. CI on 4b483a3 will show whether it recurs.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I rechecked head 4b483a35 after your Nov 12 fixes. The prior P1 block (close/merge compensating publish, purge archive/disk/slug-live, close dual-bucket probes, sweep test) looks landed; deny harness 53/53 passes locally. Two low-severity items remain on this PR’s write-brake work.

Merge readiness

  • GitHub still shows CHANGES_REQUESTED on this PR until a fresh approving review lands on 4b483a35.
  • Required Rust jobs on the latest run are green (fmt, clippy, MSRV, release build, deny-related unit surface); test (stable) / test (beta) were still pending at review time — confirm they finish green on 4b483a35 (you noted an out-of-diff reconciliation flake on the prior head).
  • Branch is mergeable with current main; no conflict blocker observed.
  • Slug-normalization / mirror-ingest follow-ups you deferred to a dedicated issue are acknowledged and are not re-listed here as PR-scope defects.

Findings

🔵 P3 — close_issue can still debit write_rate_limiter without closing when both buckets look open

📍 Where: crates/gitlawb-node/src/api/issues.rs (close_issue, ~298–306)

💥 What fails: After both is_throttled probes pass, write_rate_limiter.check runs before close_issue_rate_limiter.check. If the write check succeeds and the close check then fails (close already at cap due to a concurrent close), the write slot is consumed but no close runs.

🔎 Root cause: Sequential consuming checks after non-consuming probes; the second failure does not roll back the first debit.

📜 Stated contract:

In-handler comment: a close refused by either cap spends nothing from the other bucket.

🏷️ Attribution: PR-introduced (dual-bucket + probe pattern on this branch).

📌 In this PR:

  • close_issue rate-limit gate only

🔒 Unchanged on main: Outer write_braked placement on other routes.

🔧 Required correction: Debit both buckets only when both checks will succeed (e.g. paired non-consuming admission then paired check, or restore write slot if close check fails), and extend tests beyond the fully-saturated-bucket case to cover write-ok/close-fail ordering.

🛠️ Author fix: Keep INV-12 read-gate-first ordering; fix only the dual-bucket debit logic and tests.

🚫 Out of scope: Re-mounting issue routes with router-level write_braked.


🔵 P3 — New in-handler write_rate_limiter debits omit Retry-After while middleware 429s advertise window delay

📍 Where: crates/gitlawb-node/src/api/issues.rs (create_issue_comment ~185–190; close_issue write-bucket arms ~298–306)

💥 What fails: These paths return AppError::TooManyRequests with no retry-after header. This PR’s router brakes use check_retry and attach window-derived Retry-After on 429.

🔎 Root cause: Handler paths call check() (bool) instead of check_retry, and TooManyRequests maps to status-only.

📜 Stated contract:

This PR extended limiter and middleware 429 behavior to return and test window-derived Retry-After (rate_limit.rs, receive-pack advert and middleware tests).

🏷️ Attribution: PR-introduced incomplete — only the write_rate_limiter debits added or wired on this branch for issue comment/close; the pre-existing close_issue_rate_limiter-only path at merge-base already omitted Retry-After and is not part of this finding.

📌 In this PR:

  • create_issue_comment post-read-gate write debit
  • close_issue shared write-bucket debit (not the close-only cap alone)

🔒 Unchanged on main: Historical handler 429 sites that never debited write_rate_limiter; global redesign of every TooManyRequests variant.

🔧 Required correction: On those write-bucket rejections, use check_retry (or return the same too_many_requests(Duration) helper middleware uses) so 429 carries retry-after; one regression on comment or close write-cap is enough.

🛠️ Author fix: Local helper is fine; do not retrofit unrelated handlers in this PR.

🚫 Out of scope: Adding Retry-After to merge-base close-cap-only 429s or other untouched limiters.

…ry-After on write-bucket 429s (#196)

close_issue debited the write bucket and then the close bucket, so a
concurrent close taking the last close slot between the probe and the
check left the write slot spent with no close done. Drop the probes:
debit write, then close, and refund the write slot when close refuses.

The in-handler write-bucket refusals in close_issue and
create_issue_comment now return AppError::RateLimited, which maps to 429
with a window-derived Retry-After like the router middleware's 429s. The
close-cap-only refusal keeps its plain 429.
@beardthelion

Copy link
Copy Markdown
Collaborator Author

close_issue can still debit write_rate_limiter without closing when both buckets look open

Fixed in 11cadb9. I dropped the probes and close_issue now debits the write bucket, then the close bucket, and refunds the write slot (new RateLimiter::refund) when the close cap refuses. That covers the concurrent-close window you described, which the probes could not. close_issue_refused_by_close_cap_does_not_debit_write_bucket fails with the refund line removed and passes with it, and the reverse-direction test still passes.

New in-handler write_rate_limiter debits omit Retry-After while middleware 429s advertise window delay

Fixed in 11cadb9. The write-bucket refusals in create_issue_comment and close_issue now go through check_retry and return a new AppError::RateLimited(Duration), which maps to 429 with an integer Retry-After taken from the window. close_issue_write_cap_429_carries_retry_after and create_issue_comment_write_cap_429_carries_retry_after both failed before the change (no header) and pass now. The close-cap-only 429 is unchanged, per your out-of-scope note.

@jatmn jatmn left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Thanks for the contribution. I do not see any actionable issues from my review.

Merge readiness

Head 11cadb938260dc8d6ddd2310e3930014a4e5ddb0 is mergeable but two commits behind live origin/main (5a7576b682e1fc2566c25b09a1b69edf85e836cb). Rebase onto current main before merge so any recent main fixes ride with this branch.

Required test (stable) and test (beta) failed on run 43337356159:

  • reconciliation::tests::sweep_persists_last_attempted_not_planned_tail_on_mid_batch_narrow (stable)
  • ipfs_pin::tests::discovery_record_elapsed_leaves_neither_row (beta)

Neither crates/gitlawb-node/src/reconciliation.rs nor crates/gitlawb-node/src/ipfs_pin.rs appears in this PR’s three-dot diff. Treat these as required-check failures that must be green on the PR head (re-run after rebase, or chase flake/regression on main if they reproduce there). Other required checks on that run (fmt+clippy, deny harness job aside from the single beta failure, MSRV, docker smoke, CodeQL matrix) were green.

Prior jatmn review items from 2026-11-11 (close/merge compensating upload rollback, purge archive-after-disk-refusal, execute-time symlink recheck, write-limiter sweep regression) are addressed on current head per code inspection and the author’s follow-up commits through 11cadb93 (including write-bucket Retry-After on in-handler refusals and close-cap refund for the single-request dual-bucket window).

This was referenced Nov 13, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

crate:node gitlawb-node — the serving node and REST API kind:feature New capability or surface subsystem:identity DID/UCAN, http-sig auth, push authorization

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants